Tuesday, June 10, 2014

Create Graph with DB Data using Chart JS - Humidity & Temperature Web Logger

Let's modify WSDHT1.0.html little bit to create graph.



There is nothing much different from other previous graphs, except that we have two Data and we have two vertical axis.

First do Chart Configuration:

        $(function() {
            $("#chartContainer").dxChart({
                title: {
                    text: 'Temperature & Humidity Graph'
                },
                // Left y-axis: temperature, right y-axis: humidity
                valueAxis: [{ name: 'mainAxis', position: 'left', title: { text: 'Temperature (C)'} },
                        { name: 'secondaryAxis', position: 'right', title: { text: 'Humidity (%)'}}],
                dataSource: chartDataSource,
                legend: { verticalAlignment: "bottom", horizontalAlignment: "center" },
                // Humidity & Temperature have common x-axis: time
                commonSeriesSettings: { argumentField: 'time' },
                // Use 'spline' type to make smooth curves
                series: [{ name: 'Temperature', valueField: 'temperature', point: { visible: false }, type: 'spline', axis: 'mainAxis' },
                     { name: 'Humidity', valueField: 'humidity', point: { visible: false }, type: 'spline', axis: 'secondaryAxis'}]
            });
        });


Then Add Chart Data:

        function graphData() {
            var len, i;
            var chartData = [];
            blockNo = document.getElementById("DataBlockNo").value;
            db.transaction(function(tx) {
                tx.executeSql('SELECT * FROM LogData WHERE block="' + blockNo + '"', [], function(tx, results) {
                    len = results.rows.length;
                    if (len > 42) len = 42;
                    for (i = 0; i < len; i++) {
                        chartData.push({ time: i, temperature: parseFloat(results.rows.item(i).temperature), humidity: parseFloat(results.rows.item(i).humidity) });
                    }
                    observableChartDataSource = ko.observableArray(chartData);
                    var chart = $("#chartContainer").dxChart("instance");
                    var dataSource = chart.option('dataSource', chartData);
                }, null);
            });
        }


Full codes can be found in following links





Monday, June 9, 2014

Log Data Upload - Humidity & Temperature Web Logger

I have purchased DHT11 Humidity & Temperature Sensor from the following site.
http://www.dfrobot.com/index.php?route=product/product&product_id=174#.U4lzu3J_uSq

They deliver only the sensor.  It has Connector and Wire; however, there is no PCB.



BUT, it doesn't have cable to connect to Arduino.
So I had to purchase the Arduino jumper wire set, which I thought was useless. :(
http://www.eleparts.co.kr/EPXCTKGY ( the following site is Korean site. You might have to search for other site in your own country if you need to purchase one. )



Using the correct colors ( it's not a must ), connect it to Arduino.  I attached it to Connector on RTC Shield.

Red cable: +5V Power.  Top left connector, fifth pin from the top.
Black cable: GND.  Bottom left connector, first pin from the top.
Green cable: A0.  Top right connector, fourth pin from the top.

Download library and import it.


If you upload the dht.ino included in the zip file,



  1. Temperature: It was very hot today; however, it looks somewhat correct.  When I get an actual thermometer I will re-check.
  2. Humidity: Humidity calculation is based on Temperature, thus if the temperature is not correct humidity cannot be correct.
  3. Although there is a decimal point, the decimal value is always 0.  If we change to better sensor I assume it will give us the decimal values so I'm going to stick with the float.  Accuracy of sensor is not the point in this posting so I'm going to move on.

In dht.ino change #define DHT11_PIN 4 to #define DHT11_PIN A0, and 
chk = DHT.read11(DHT11_PIN); to int chk = DHT.read11(DHT11_PIN);.


Protocol

I have modified Get Data Block "GETn".
D = <time stamp> + <Temperature> + <Humidity>
We are going to send time stamp, temperature, and humidity all at the same time.



Arduino Code

Following is excerpts from the code.  The entire code can be found in the link at the very bottom.  I modified WSAdam5.ino to create the following code.
#include <dht.h>
... OMIT ...
#define LOG_BLOCK_SIZE 96
... OMIT ...
//-------------------------------------DHT-11
#define DHT11_PIN A0
//#define SAMPLING_INTERVAL (3*1000)

// If you set it to this, it will read sensor value every minute.
// It takes forever to debug if you set it to 10 minutes.
// Arduino default is 16 bit int. If you don't do (uint32_t)
// it becomes -5536
#define SAMPLING_INTERVAL (uint32_t)60*1000
#define MAX_REC_SIZE  42 // 512/12 = 42.666...

dht DHT;
float humidity;
float temperature;
void DHTPoll();
... OMIT ...
void loop() {
 wsServer.listen();
 
 if(sdStatus) {
  if((millis() - lastPollTime) >= SAMPLING_INTERVAL) {
   lastPollTime = millis();
   DHTPoll();
  }
 }
}

/*
  ----------------------------------------------------------------
   D H T - 1 1   h a n d l e r
  ----------------------------------------------------------------
*/
void DHTPoll() {
 if(DHT.read11(DHT11_PIN) == DHTLIB_OK) {
  humidity = DHT.humidity;
  temperature = DHT.temperature;

  if(sdStatus) makeLogBlock();
 }
}
... OMIT ...
void SDbegin(void) {
... OMIT ...
 //load last block
 sdCard.readBlock(currBlock, sdBuffer);
 pB = sdBuffer;

 if(*((uint32_t*)pB) != USED_MARK) { // New data block
  *((uint32_t*)pB) = USED_MARK; // Used Flag
  *((uint32_t*)(pB + 4)) = 0; // for this Logging Block
                              // set the record to 0 and
                              // add 3 bytes padding
 }

 LogBlockSize = sdBuffer[4]; // Used data block
 sdStatus++;  
 printCurrBlock(); //debug purpose
}
... OMIT ...
void makeLogBlock(void) {
... OMIT ...
 //------------------------- push Data into Buffer
 pB = sdBuffer + 8 + (LogBlockSize * 12);
 now = rtc.now();

 *((uint32_t*)pB) = now.unixtime(); // Time-stamp 
 *((float*)(pB + 4)) = temperature; // Temperature
 *((float*)(pB + 8)) = humidity; // Humidity
... OMIT ...
 //------------------------- write to SD card everytime
 LogBlockSize++;
 sdBuffer[4] = LogBlockSize;
 sdCard.writeBlock(currBlock, sdBuffer);

 //------------------------- If the Data Block is complete, 
 //           fix the Free block location and reset Buffer
 if(LogBlockSize >= MAX_REC_SIZE) {
  currBlock++;
  update_currBlock();

  pB = sdBuffer;
  *((uint32_t*)pB) = USED_MARK; // Used Flag
  *((uint32_t*)(pB + 4)) = 0; // Set this logging Block's record
                              // to 0 and add 3 bytes padding
  LogBlockSize = 0;
 }  
... OMIT ...
void upload_DataBlock(uint32_t reqBlock) {
... OMIT ...
 cnt = *pS;
 pS += 4;
 printLn[0] = 'D';

 while(cnt--) {
  // time stamp (seconds)
  ltoa(*((uint32_t*)pS), printLn + 2, 10);
  pC = printLn + strlen(printLn);
  *pC++ = '=';
  pS += 4;
  
  // Temperature
  dtostrf(*((float*)pS), 6, 2, pC); // -12.34C
  pC = printLn + strlen(printLn);
  *pC++ = '=';
  pS += 4;

  // Humidity
  dtostrf(*((float*)pS), 6, 2, pC); // 12.34%
  pS += 4;
  WSsend(printLn, strlen(printLn));
 }

 printLn[2] = '0'; //timestamp == 0, end-of-data
 WSsend(printLn, 3);
... OMIT ... 


HTML / JS Code

This code shows how to upload data using Web Socket, and how to create Web DB using the uploaded data.  Codes below are only part of the code.  If you want to see the entire code look at the link at the very bottom.
This code is modified from WSAdam6_5.html.

Clicking [GetDataBlock] button will display the Temperature & Humidity Data in DB.



... OMIT ...
      var db = openDatabase('WebLogger10', '1.0', 'Web Logger', 512*1024);

      $(document).ready(function() {
        WebSocketConnect();
        db.transaction(function (tx) {
          tx.executeSql('CREATE TABLE IF NOT EXISTS LogIndex(block, timeStamp)');
          tx.executeSql('CREATE TABLE IF NOT EXISTS LogData(block, timeStamp, temperature, humidity)');
        });
      });
... OMIT ...
      function WebSocketConnect() {
... OMIT ...
            // receive Data Block
            if(ar[0] == 'D'){
              data[rcvCnt++] = ar[1];
              data[rcvCnt++] = ar[2];
              data[rcvCnt++] = ar[3];
              if(ar[1] == '0'){
                $("#recordCntPara").empty();  
                $("#recordCntPara").append('Received:' + rcvCnt);
                if(blockNo != 0){
                  db.transaction(function (tx) {                  
                    if(rcvCnt>126) rcvCnt = 126;
                    rcvCnt /= 3;
                    $("#recordCntPara").append(',Record Count:' + rcvCnt);
                    for(i=0;i<rcvCnt;i++){
                      tx.executeSql("INSERT INTO LogData VALUES(?, ?, ?, ?)", [blockNo, data[i*3], data[i*3+1], data[i*3+2]]);
                    } 
                  });
                  db.transaction(function (tx) {
                    tx.executeSql("INSERT INTO LogIndex VALUES(?, ?)", [blockNo, data[0]]);
                  });
... OMIT ...
      function viewData() {
... OMIT ...
            for(i=0;i<len;i++){
              timeStamp = results.rows.item(i).timeStamp;
              tod.setTime((timeStamp - (9*3600)) * 1000);
              $("#dumpPara").append('<p>' + tod.toLocaleString() + '  ' + results.rows.item(i).temperature + 'C ' + results.rows.item(i).humidity + '%<p>');
            }
... OMIT ...



Full codes can be found in following links





Sunday, June 8, 2014

Construct - Humidity & Temperature Web Logger

This time I'm going to create Humidity & Temperature Logger to outline how easy it is to create Web Logger.



Humidity & Temperature Sensor

I have used DHT11, the most common Arduino Humidity & Temperature sensor.
http://www.dfrobot.com/index.php?route=product/product&product_id=174#.U4--irkU9aR
The above site has many more interesting gadgets to look at.

DHT11 datasheet can be downloaded from following site, or it can also be found in the link at the bottom.
http://www.micro4you.com/files/sensor/DHT11.pdf

Item Measurement Range Humidity Accuracy Temperature Accuracy Resolution Package
DHT11 20-90% RH
0-50°C
±5% RH ±2°C 1 4 Pin Single Row

Accuracy for both Temperature & Humidity sucks; however, DHT11 is easy to purchase and connect.  I will try working with higher accuracy model next time.

It says Resolution for both are 8 bit; however, if you look at the data format it is a different story.

Data Format: 

  • 8 bit integral RH data 
  • + 8 bit decimal RH data 
  • + 8 bit integral T data 
  • + 8 bit decimal T data 
  • + 8 bit check sum

If we start thinking about decimal points it gets confusing when coding.  Thus we are going to use floating just like we did for ADAM-4017.




Data Block Components

It is highly unlikely Temperature & Humidity changes drastically.
For this sensor, the response time for humidity is 15 seconds and temperature is 30 seconds ( in data sheet ).
I am going to measure every 10 minutes.

If we measure every 10 minutes, it is going to take multiple hours to fill up 1 sector.  And if for some reason if you decide to reboot, you are going to lose hours worth of Data.
This is very unwanted situation, thus we are going to write to SD every time we measure.
Add time stamp to each measured data.


  • Used Flag: 4 bytes
    Set it to 0xaa55a55b
  • Data Count: 1 byte
    Number of Data in sector. 0, 1 - 42
  • Extra: 3 bytes
    Fill it with Null.
  • Data: 12 bytes
    42 will fit in 1 Data Block (8 + 42 * 12 = 512)
    • Time Stamp: 4 bytes
      Use only sec part of POSIX time_t.
      Must *1000 because we are not using millisecond part.
      It is 32 bit unsigned integer.
    • Temperature: 4 bytes floating
    • Humidity: 4 bytes floating

Each sector will contain 42 * 10 min = 420 min = 7 hrs worth of data.
Each Data has its own time stamp, thus sector is not ordered in time or date.



Full codes can be found in following links




Saturday, June 7, 2014

RTC Connection Code - Arduino Web Logger

Add the protocol to check and modify RTC time.

Get RTC Time

When you send "RTC0", it replies with T=<time stamp>
<time stamp> is only the sec part of POSIX time_t.

Set RTC Time

When you send "RTC<time stamp>", it replies with T=<time stamp>


Original Protocol

Get Last Data Block Number

When you send "GET0", it replies with N=<Data Block Number>

<Data Block Number>: "N=1" - "N=4294967295" unsigned 32 bit integer.
Of course, it will not get this big.

In the Data Log Code - Arduino Web Logger posting source code, SD_BASE_BLOCK becomes 0.

Get Data Block

When you send "GETn", it replies with
  • N = <Data Block Number>
  • T = <time stamp>
  • D = <data>
<time stamp> is only the sec part of POSIX time_t.  Do *1000 since we don't use millisecond part.  Of course it will be very difficult to see "T=4294967295" in our lifetime.

<data> is in +dd.ddd format.
  • "D=+12.345"
  • "D=" no data
  • "D=-99.999" termination.  That certain Data Block send is completely finished.

Digital Out

When turning on "DOn=ON",
when turning off "DOn=OFF".

n is channel number between 0 to 7.  There is no reply.

Digital In

When you send "DI", it replies with "bbbbbbbb".
b represents each input channel, and it becomes either '0' or '1'.

Analog In

When you send "AIn", it replies with "+dd.ddd".
n is channel number between 0 and 7, and "+dd.ddd" is from -10.000 to +10.000.


When you click [GetRTC] button display RTC time.



The incorrect RTC time gets set to current time when [SetRTC] button is clicked.




Now let's look at modified code.  ( Look at link below for complete code )

Fixed code from Arduino

#include <SPI.h>
#include <Ethernet.h>
#include <WebSocket.h>
#include <SoftwareSerial.h>
#include "Sd2Card.h"
#include <minWire.h> // use the modified library code from RTC Preparation posting
#include <minRTClib.h>

// ... OMIT ...

//-------------------------------------DS-1307
RTC_DS1307 rtc;

// ... OMIT ...

//------------------------------------------------ setup()
void setup() {
 Serial.begin(9600);
 Wire.begin();
 rtc.begin();

 // ... OMIT ...

//-----------------------------------------------
// Create DATA Logging Block
//-----------------------------------------------
void makeLogBlock(void) {
 byte i;
 byte *pB;
 float fdata;
 DateTime now;

 // ... OMIT ...

 //------------------------- buffer is full
 if(LogBlockSize >= LOG_BLOCK_SIZE) {
  pB = sdBuffer;
  *((uint32_t*)pB) = USED_MARK;  //Used Flag
  pB += 4;
  now = rtc.now();
  *((uint32_t*)pB) = now.unixtime(); //Time-stamp

  // ... OMIT ...

void printTOD(void) {
 DateTime now = rtc.now();
 Serial.print(now.year(), DEC);
 printProg(DASH_MSG);
 Serial.print(now.month(), DEC);
 printProg(DASH_MSG);
 Serial.print(now.day(), DEC);
 printProg(SPACE_MSG);
 Serial.print(now.hour(), DEC);
 printProg(COMMA_MSG);
 Serial.print(now.minute(), DEC);
 printProg(COMMA_MSG);
 Serial.println(now.second(), DEC);
}

// ... OMIT ...

void onData(WebSocket &socket, char* rxLn, byte rxSz) {
 uint32_t para;
 DateTime now;

 //-----------------------------------
 // GET
 //-----------------------------------
 if((rxLn[0] == 'G') && (rxLn[1] == 'E') && (rxLn[2] == 'T')) {
  *(rxLn + rxSz) = 0x00;
  para = atol(rxLn + 3);
  
  if(para) 
   upload_DataBlock(para + SD_BASE_BLOCK);
  else 
   upload_lastBlockNo();
 }

 //-----------------------------------
 // RTC
 //-----------------------------------
 else if((rxLn[0] == 'R') && (rxLn[1] == 'T') && (rxLn[2] == 'C')) {
  *(rxLn + rxSz) = 0x00;
  para = atol(rxLn + 3);

  // set RTC
  if(para) {
   rtc.adjust(DateTime(para));
   delay(100);
  }

  // send current RTC
  printLn[0] = 'T';
  printLn[1] = '=';
  now = rtc.now();
  ltoa((long int)(now.unixtime()), printLn+2, 10);
  WSsend(printLn, strlen(printLn));
 }
}


Fixed code from HTML/JS

        // ... OMIT ...
        
        // receive Time Stamp
        if(ar[0] == 'T') {
            timeStamp = ar[1];
            // Locale standard is GMT. 
            // In order to view as Korean time, we need to -9 hrs.
            tod.setTime((timeStamp - (9 * 3600)) * 1000); 
            
    // ... OMIT ...
    
    function getRTC() {
        clearDspPara();
        ws.send('RTC0');
    }
    
    function setRTC() {
        clearDspPara();
        var t = new Date();
        // GMT problem. Ths time +9 hrs.
        ws.send('RTC' + (t / 1000 + 9 * 3600));
    }
    
// ... OMIT ...

<button type="button" onclick="getRTC()">Get RTC</button>
<button type="button" onclick="setRTC()">Set RTC</button>

// ... OMIT ...



Full codes can be found in following links





Friday, June 6, 2014

RTC Preparation - Arduino Web Logger

This is somewhat out of order, BUT let's connect RTC.

I purchased RTC Shield from following site.
https://store.open-electronics.org/index.php?_route_=RTC%20shield

Then the following gets delivered.


They gave me extra connector! :)

Also it comes with manual that I cannot read.  Italian maybe...?


Fortunately, there is no problem reading the schematic.


After some soldering...




Now we have four story high building.
We have on the first floor Arduino, second floor Ethernet Shield, third floor RTC Shield, and finally on the fourth floor we have RS485 Shield.




The library for RTC can be downloaded in the following site.
You might have to modify the name of the folder after downloading it.  Arduino sketch won't let you import 'RTClib-master', thus fix the name to 'RTClib_master'.

However, there is a problem.  This library takes up too much memory.
This library brings our memory down to 170 bytes ( before this library we have 377 bytes ).  We are not able to use this like this.

Let's look through, and find who uses up the most memory.

1. We are only going to use IIC Master mode and not the Slave.  So let's get rid of all Slave part.


2. Reduce memory from writing to RTC part.


RTClib.cpp
void RTC_DS1307::adjust(const DateTime& dt) {
 WIRE.beginTransmission(DS1307_ADDRESS);
 WIRE.write(0);
 WIRE.write(bin2bcd(dt.second()));
 WIRE.write(bin2bcd(dt.minute()));
 WIRE.write(bin2bcd(dt.hour()));
 WIRE.write(bin2bcd(0));
 WIRE.write(bin2bcd(dt.day()));
 WIRE.write(bin2bcd(dt.month()));
 WIRE.write(bin2bcd(dt.year() - 2000));
 WIRE.write(0);
 WIRE.endTransmission();
} // The import thing is that the maximum length is 9 bytes

Wire.cpp
size_t TwoWire::write(uint8_t data) {
 // ... OMIT ...
 txBuffer[txBufferIndex] = data; // size of txBuffer[] is 
                                    // #define BUFFER_LENGTH 32 in
                                 // Wire.h
                                    // Reduce to size 9
 ++txBufferIndex;
 txBufferLength = txBufferIndex;

// ... OMIT ...

uint8_t TwoWire::endTransmission(uint8_t sendStop) {
 int8_t ret = twi_writeTo(txAddress, txBuffer, txBufferLength, 1, sendStop);

twi.c
uint8_t twi_writeTo(uint8_t address, uint8_t* data, uint8_t length, uint8_t wait, uint8_t sendStop) {
 // ... OMIT ...

 for(i = 0; i < length; ++i) {
  // Because ISR sends actual IIC, it has it's own Buffer.
  // If you make sure that you won't send until it is
  // completely finished, then you don't need that Buffer.
  twi_masterBuffer[i] = data[i];
 }

 // ... OMIT ...


3. Reduce memory from reading from RTC part.


RTClib.cpp
DateTime RTC_DS1307::now() {
 WIRE.beginTransmission(DS1307_ADDRESS);
 WIRE.write(0); 
 WIRE.endTransmission();
 WIRE.requestFrom(DS1307_ADDRESS, 7); // Receiving 7 bytes
 uint8_t ss = bcd2bin(WIRE.read() & 0x7F);
 uint8_t mm = bcd2bin(WIRE.read());
 uint8_t hh = bcd2bin(WIRE.read());
 WIRE.read();
 uint8_t d = bcd2bin(WIRE.read());
 uint8_t m = bcd2bin(WIRE.read());
 uint16_t y = bcd2bin(WIRE.read()) + 2000;
 return DateTime (y, m, d, hh, mm, ss);
}

Wire.cpp
uint8_t TwoWire::requestFrom(uint8_t address, uint8_t quantity, uint8_t sendStop) {
 // ... OMIT ...

 // Reduce the size of rxBuffer[] to 7
 uint8_t read = twi_readFrom(address, rxBuffer, quantity, sendStop);

 // ... OMIT ...
}

twi.c
uint8_t twi_readFrom(uint8_t address, uint8_t* data, uint8_t length, uint8_t sendStop) {
 //... OMIT ...

 for(i = 0; i < length; ++i){
  // Received from ISR. There is no actual benefits for
  // moving to pointer, so let's keep 2 buffer.
  // fix the size of twi_masterBuffer[] to 7
  data[i] = twi_masterBuffer[i];
 }

 // ... OMIT ...
}


We have reduced memory to 142 bytes.

I have uploaded the modified library with "min" added to the original name.
Import the attached library.

Full codes can be found in following links






Thursday, June 5, 2014

Drawing DB Data with Chart JS - Arduino Web Logger



This time we are going to draw graph with the data we have seen as text.
I have used Spline Charts.  I chose this graph because it has smooth curve lines.
http://chartjs.devexpress.com/Demos/VizGallery/#chart/chartslineandpointseriesspline
Refer to below site for how to make chart.
http://chartjs.devexpress.com/Documentation/Tutorial/Configure_Charts?version=13_2

Use the code in Web SQL Database - Arduino Web Logger posting after deleting upload part and inserting DB part.

For detailed information about how to use Chart JS refer to HTML / JavaScript - 4017 Arduino Gateway posting.  Include knockout-3.0.0.js this time.

<!DOCTYPE html>
<html>
    <head>
        <title>WebLogger 7.1</title>
        // ******************** 
        //  Below js library are in
        //  DevExpressChartJS-13.2.9\Lib\js.
        // ********************
        <script src="jquery-2.0.3.min.js"></script>
        <script src="globalize.min.js"></script>
        <script src="knockout-3.0.0.js"></script>
        <script src="dx.chartjs.js"></script>
        
        <script>
            var blockNo;
            var data = new Array();
            var tod = new Date();
            var db = openDatabase('WebLogger8', '1.0', 'Web Logger', 512 * 1024);

            $(document).ready(function() {
                db.transaction(function(tx) {
                    tx.executeSql('CREATE TABLE IF NOT EXISTS LogIndex(block, timeStamp)');
                    tx.executeSql('CREATE TABLE IF NOT EXISTS LogData(block, seq, data)');
                });

            });

            // ********************
            // Create Chart.
            // ********************
            var chartDataSource = [];

            $(function() {
                $("#chartContainer").dxChart({
                    title: {
                        text: 'Logged Data Graph'
                    },
                    dataSource: chartDataSource,
                    commonSeriesSettings: {
                        type: 'spline',
                        argumentField: 'time'
                    },
                    series: [{ name: 'ADAM-4017', valueField: 'data', point: { visible: false}}]
                });
            });

            function viewIndex() {
                var len, i;

                db.transaction(function(tx) {
                    tx.executeSql("SELECT * FROM LogIndex", [], function(tx, results) {
                        len = results.rows.length;
                        clearDspPara();

                        for (i = 0; i < len; i++) {
                            tod.setTime(results.rows.item(i).timeStamp * 1000);
                            $("#dumpPara").append('<p>' + results.rows.item(i).block + ',' + tod.toLocaleString() + '<p>');
                        }
                    }, null);
                });
            }

            function viewData() {
                var len, i;
                blockNo = document.getElementById("DataBlockNo").value;

                db.transaction(function(tx) {
                    tx.executeSql('SELECT * FROM LogData WHERE block="' + blockNo + '"', [], function(tx, results) {
                        len = results.rows.length;

                        if (len > 120) len = 120;

                        clearDspPara();

                        for (i = 0; i < len; i++) {
                            data[results.rows.item(i).seq] = results.rows.item(i).data;
                        }

                        for (i = 0; i < len; i++) {
                            $("#dumpPara").append('<p>' + data[i] + '<p>');
                        }
                    }, null);
                });
            }

            function graphData() {
                var len, i;
                var chartData = [];
                blockNo = document.getElementById("DataBlockNo").value;

                db.transaction(function(tx) {
                    tx.executeSql('SELECT * FROM LogData WHERE block="' + blockNo + '"', [], function(tx, results) {
                        len = results.rows.length;

                        if (len > 120) len = 120;

                        // ********************
                        //  Create Data for Chart
                        // ********************
                        for (i = 0; i < len; i++) {
                            // If string goes into chart data
                            // everything screws up. Here I 
                            // converted string to float.
                            // More about conversion:
                            // http://www.javascripter.net/faq/convert2.htm
                            data[results.rows.item(i).seq] = parseFloat(results.rows.item(i).data);
                        }

                        for (i = 0; i < len; i++) {
                            chartData.push({ time: i * 0.5, data: data[i] });
                        }

                        observableChartDataSource = ko.observableArray(chartData);
                        var chart = $("#chartContainer").dxChart("instance");
                        var dataSource = chart.option('dataSource', chartData);
                    }, null);
                });
            }

            function clearDspPara() {
                $("#dumpPara").empty();
            }
        </script>
    </head>
    
    <body>
        <!--Create space for Chart-->
        <div id="chartContainer" style="width: 720px; height: 440px;"></div>
        <br />
        
        <button type="button" onclick="viewIndex()">View Index</button>
        <input type="text" id="DataBlockNo" value="1">
        <button type="button" onclick="viewData()">View Data</button>
        <button type="button" onclick="graphData()">Graph Data </button>
        <p id="dumpPara"></p>
    </body>
</html>


Full codes can be found in following links





Wednesday, June 4, 2014

Web SQL Database - Arduino Web Logger

The most basic HTML5.0 - Web SQL Database Tutorial.
http://www.tutorialspoint.com/html5/html5_web_sql.htm

1. Set DB's Name and Size.

    var db = openDatabase('WebLogger8', '1.0', 'Web Logger', 512 * 1024);

DB's name is WebLogger8 and the size is 512K bytes.

2. Create 2 DB Table. One with Index and the other with Data.

    db.transaction(function(tx) {
        tx.executeSql('CREATE TABLE IF NOT EXISTS LogIndex(block, timeStamp)');
        tx.executeSql('CREATE TABLE IF NOT EXISTS LogData(block, seq, data)');
    });

  • LogIndex: this is an index table with 2 columns
    1. block number &
    2. timeStamp
  • LogData: this is an data table
    1. block number,
    2. index within the block (500 ms = index 0, 1000 ms = index 1...)
    3. actual data value

3. Add Data Block to DB Table.

    db.transaction(function(tx) {
        for (i = 0; i < rcvCnt; i++) {
            tx.executeSql('INSERT INTO LogData VALUES(?, ?, ?)', [blockNo, i, data[i]]);
        }
    });

Append the data table.  You must write to DB after you have received entire Data Block.  If you try to write data values every time you receive it, you will lose a lot of data.  The confusing thing is that you receive correct number of data; however, you don't see the actual values anywhere.

We don't define data type for each column.  This makes things simpler and confusing at the same time.  When you actually write to DB the type gets defined.  I am assuming that block becomes String, seq becomes Number, and data becomes String.

    db.transaction(function(tx) {
        tx.executeSql('INSERT INTO LogIndex VALUES(?, ?)', [blockNo, timeStamp]);
    });

Lastly append the index.
block and timeStamp are both String type.

Caution:
    db.transaction(function(tx) {
        tx.executeSql('INSERT INTO LogData VALUES(?, ?, ?)', [blockNo, i, data[i]]);
        tx.executeSql('INSERT INTO LogIndex VALUES(?, ?', [blockNo, timeStamp]);

Inside one db.transaction, do not append DB twice.  Theoretically this is not a problem; however, if you continue to use that statement it causes error so let's avoid it.

4. Check DB Table.

    db.transaction(function(tx) {
        // Check all index.
        tx.executeSql('SELECT * FROM LogIndex', [], function(tx, results) {
            len = results.rows.length;

            for (i = 0; i < len; i++) {
                // we don't use millisecond part thus do *1000
                tod.setTime(results.rows.item(i).timeStamp * 1000);
                // if you do toLocaleString() to
                // day-time format variable, it returns a string
                // with a language sensitive representation of this
                // date.  There are many others. 
                // http://www.w3schools.com/jsref/jsref_obj_date.asp
                $("#dumpPara").append('' + results.rows.item(i).block + ',' + tod.toLocaleString() + '

');
            }
        }, null);
    });

    db.transaction(function(tx) {
        // search DB with block
        tx.executeSql('SELECT * FROM LogData WHERE block="' + blockNo + '"', [], function(tx, results) {
            len = results.rows.length;

            for (i = 0; i < len; i++) {
                // There is no guarantee that DB will read in order we
                // wrote. Thus we index it inside block.
                data[results.rows.item(i).seq] = results.rows.item(i).data;
            }

            for (i = 0; i < len; i++) {
                $("#dumpPara").append('

' + data[i] + '

');
            }
        }, null);
    });






5. For Less Confusion while Debugging...

  1. When you try to change the name of DB, close all Chrome windows and re-open it.  Otherwise new DB Table does not get created.
  2. If HTML file is not in the same Folder path, close all Chrome windows and re-open it.  If you don't do this WebSocket doesn't get connected.

Get the last SD logged Block Number from Arduino.

Click the [Get Last Data Block Number] button.



Enter the Block Number and Upload that certain Data Block from Arduino SD.

Enter the Block Number and click [Get Data Block] button.

If you upload, it gets inserted into both Index table and Log table.


Check Index Table.

Click [View Index] button.
It is displayed in Block, Time Stamp order.


Enter Data Block number to check Data Table.

Enter Block Number and click [View Data] button.

You will be able to see 120 values from that Block.  Only the data will be shown.


Below is HTML/JS Code

The link to the code is posted in the end.
<!DOCTYPE html>
<html>
    <head>
        <title>WebLogger 6.3</title>
        <script type="text/javascript" src="jquery-2.0.3.min.js"></script>
        <script>
            var ws;
            var blockNo, rcvCnt, timeStamp;
            var data = new Array();
            var tod = new Date();

            var db = openDatabase('WebLogger8', '1.0', 'Web Logger', 512 * 1024);

            $(document).ready(function() {
                db.transaction(function(tx) {
                    tx.executeSql('CREATE TABLE IF NOT EXISTS LogIndex(block, timeStamp)');
                    tx.executeSql('CREATE TABLE IF NOT EXISTS LogData(block, seq, data)');
                });

                WebSocketConnect();
            });

            function WebSocketConnect() {
                var ar = new Array();
                var arSz, i;

                try {
                    ws = new WebSocket('ws://192.168.219.16:80/');
                    ws.onopen = function() {
                        status('Connected...');
                    }

                    ws.onclose = function() { status('Closed...'); }
                    ws.onerror = function(evt) { status('Error ' + evt.data); }
                    ws.onmessage = function(evt) {
                        ar = evt.data.split('=');

                        // receive Last Data Block Number
                        if (ar[0] == 'N') {
                            blockNo = ar[1];
                            $("#blockNoPara").empty();
                            $("#blockNoPara").append('Block Number:' + blockNo);
                        }

                        // receive Time Stamp
                        if (ar[0] == 'T') {
                            timeStamp = ar[1];
                            tod.setTime(timeStamp * 1000);
                            $("#timestampPara").empty();
                            $("#timestampPara").append(tod.toLocaleString());
                        }

                        // receive Data Block
                        if (ar[0] == 'D') {
                            data[rcvCnt] = ar[1];

                            if (ar[1] != '-99.999') rcvCnt++;
                            else {
                                $("#recordCntPara").empty();
                                $("#recordCntPara").append('Record Count:' + rcvCnt);

                                if (blockNo != 0) {
                                    db.transaction(function(tx) {
                                        if (rcvCnt > 120) rcvCnt = 120;
                                        for (i = 0; i < rcvCnt; i++) {
                                            tx.executeSql("INSERT INTO LogData VALUES(?, ?, ?)", [blockNo, i, data[i]]);
                                        }
                                    });

                                    db.transaction(function(tx) {
                                        tx.executeSql("INSERT INTO LogIndex VALUES(?, ?)", [blockNo, timeStamp]);
                                    });
                                }
                            }
                        }
                    }
                } catch (exception) { status('Exception' + exception); }
            }

            function upload_lastBlockNo() {
                ws.send("GET0");
            }

            function upload_DataBlock() {
                blockNo = document.getElementById("DataBlockNo").value;
                ws.send("GET" + blockNo);
                blockNo = 0;
                rcvCnt = 0;
                $("#timestampPara").empty();
                $("#recordCntPara").empty();
                $("#dumpPara").empty();
            }

            function status(str) {
                $("#status").empty();
                $("#status").append(str);
            }

            function viewIndex() {
                var len, i;

                db.transaction(function(tx) {
                    tx.executeSql("SELECT * FROM LogIndex", [], function(tx, results) {
                        len = results.rows.length;
                        clearDspPara();

                        for (i = 0; i < len; i++) {
                            tod.setTime(results.rows.item(i).timeStamp * 1000);
                            $("#dumpPara").append('<p>' + results.rows.item(i).block + ',' + tod.toLocaleString() + '<p>');
                        }
                    }, null);
                });
            }

            function viewData() {
                var len, i;
                blockNo = document.getElementById("DataBlockNo").value;

                db.transaction(function(tx) {
                    tx.executeSql('SELECT * FROM LogData WHERE block="' + blockNo + '"', [], function(tx, results) {
                        len = results.rows.length;

                        if (len > 120) len = 120;

                        clearDspPara();

                        for (i = 0; i < len; i++) {
                            data[results.rows.item(i).seq] = results.rows.item(i).data;
                        }

                        for (i = 0; i < len; i++) {
                            $("#dumpPara").append('<p>' + data[i] + '<p>');
                        }
                    }, null);
                });
            }

            function clearDspPara() {
                $("#blockNoPara").empty();
                $("#timestampPara").empty();
                $("#recordCntPara").empty();
                $("#dumpPara").empty();
            }
        </script>
    </head>

    <body>
        <p id="status"></p>
        <button type="button" onclick="upload_lastBlockNo()">Get Last Data Block Number</button>
        <br />
        
        <input type="text" id="DataBlockNo" value="1">
        <button type="button" onclick="upload_DataBlock()">Get Data Block</button>
        <br />
        
        <button type="button" onclick="viewIndex()">View Index</button>
        <button type="button" onclick="viewData()">View Data</button>
        <p id="blockNoPara"></p>
        <p id="timestampPara"></p>
        <p id="recordCntPara"></p>
        <p id="dumpPara"></p>
        
    </body>
</html>

Full codes can be found in following links