Setting up InfluxDB and Grafana for storing and viewing Arduino measured sensor values

Arduino and its related products are undoubtedly very popular among hobbyists and even professionals. One of the areas of interest with hobbyists and professionals is interfacing with many different sensors such as temperature, humidity, light, etc. The simplest method would be storing measured values on an SD card or transmitting the data over the serial interface, but there are times where a prolonged measurement session is a requirement. In this case, storing data in a database over Serial, Ethernet or WiFi is the best solution. In this article I will talk about setting up InfluxDB on a Linux platform and how to store data in it, and how to view measurement data using Grafana.

What is InfluxDB?

InfluxDB is an open-source time series database developed by InfluxData. It is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.

InfluxDB has no external dependencies[and provides an SQL-like language with built-in time-centric functions for querying a data structure composed of measurements, series, and points. Each point consists of several key-value pairs called the fieldset and a timestamp. When grouped together by a set of key-value pairs called the tagset, these define a series. Finally, series are grouped together by a string identifier to form a measurement. Some of the InfluxDB’s features are:

• Values can be 64-bit integers, 64-bit floating points, strings, and booleans.
• Points are indexed by their time and tagset.
• Retention policies are defined on a measurement and control how data is downsampled and deleted.
• Continuous Queries run periodically, storing results in a target measurement.
• Line Protocol: Store data through HTTP, TCP or UDP protocols.

A very good point about InfluxDB for use with microcontrollers is that it does not necessarily require any timestamp as it automatically adds the time of arrival of the data (server time).

Installing InfluxDB on Linux

You should head on to InfluxDB’s download page in order to get the latest release version of InfluxDB. At the time of writing this article, the version is 1.1.1. In the download page, you will be given the chance to download the appropriate version for your Linux distribution. But  to make the stuff going faster here is the commands you need to execute to install InfluxDB (version 1.1.1):

If you are on Ubuntu or other Debian-based flavors:

$wget https://dl.influxdata.com/influxdb/releases/influxdb_1.1.1_amd64.deb
$sudo dpkg -i influxdb_1.1.1_amd64.deb

If you are on RedHat or CentOS:

$wget https://dl.influxdata.com/influxdb/releases/influxdb-1.1.1.x86_64.rpm
$sudo yum localinstall influxdb-1.1.1.x86_64.rpm

You can also install InfluxDB on Windows, but since this guide is focused on Linux, you will be on your own for that matter (it is really simple, don’t worry!)

Setting up InfluxDB on Linux

After installation, to generate the configuration file, run the following command (note that the command is influxd that is without the b at the end!):

$sudo influxd config

After running this command, there will be a config file in the ‘/etc/influxdb‘ named ‘influxdb.conf’. To change the default configuration you need to edit this file. I should remind you that this folder is write protected so you will need root access to be able to edit this file.

The most important options to check or change are listed below (you can read the exact purpose from the comments written in the config file by the developers) Make sure you uncomment the section directive as well if you are going to change it, e.g., uncomment the [admin] by removing the hash character!:

1. In the [admin] section make sure the admin page is enabled by removing the comment character ‘#‘ and setting it to ‘enabled = true‘
2. Since we are going to use the HTTP protocol to write values into InfluxDB, make sure it is also enabled under the [http] section of the config file.
3. If necessary, you can change the default port numbers for Administrator Panel (8083) and HTTP (8086).

Now in the terminal, running the command below starts the InfluxDB as a service (change start to stop to end the service!):

$sudo service influxdb start

Alternatively, if you want to see the output messages of InfluxDB live, you can start it in standalone mode (not as a service) by the following command:

$sudo influxd -config /etc/influxdb/influxdb.conf

That command starts the InfluxDB and tells it to use the configuration file that we just edited.

Now, if you open your browser and go to ‘http://localhost:8083‘ (or the another port if you have changed it in the config file), you will be greeted by the administration panel of InfluxDB.

InfluxDB folders

By default, the following folders will be created (again, read the config file’s comments for exact details) :

• /home/[username]/.influxdb => contains user specific data and meta folders to store databases.
• /var/log/influxdb => this folder will contain logs such as warnings and errors
• /var/lib/influxdb => this folder contains the variable state information, database engine, etc.
• /etc/influxdb => this folder contains configuration file(s)

The whole process in a GIF animation

At the moment, there is not an awful lot of data to see there 😉 so in the next steps we will discuss the line protocol which is the way of sending data to InfluxDB, and then we send some data from Arduino over Ethernet to InfluxDB![nextpage title=”InfluxDB Line Protocol on HTTP”]

The Scenario

Let’s assume that you have 4 Arduinos in 4 different corners of your house and you want to monitor and log the temperature, humidity and light intensity in these corners. Then for using InfluxDB for your great home environment monitoring you need only one database in InfluxDB. Let’s call the database ‘embedonix‘.

Each entry in a database is called a “Measurement” in InfluxDB terms and each “Measurement” consists of “Tag(s)” and “Value(s)”. The tags, usually being a string or an integer, are signifying a spatial concept e.g. what is the source of values and where does it come from. The values associated with tags are just the values!

So in our scenario, we need a few tags for better extraction and filtering of data from the database. These tags can be ‘SOURCE=arduino_n‘ where n is the ID you assign to your Arduinos, and another tag might be ‘PLACE=kitchen‘ or ‘PLACE=garage‘…you get the idea. Then comes the values, which for our case would be something like ‘TEMP=20‘, ‘HUMIDITY=85.7‘, ‘LIGHT=780.98‘.

The format for sending the above data would be:

[MEASUREMENT_NAME],[TAG_1=xxx],[TAG_2=xxx] [VAL_1=xxx],[VAL_2=xxx]

First, is the name of measurement then comes the tag(s) separated by a comma. After the last tag, there is a blank space, anything after this space will be treated as a value (comma separated if there is more than one value). So fixing for out scenario, an insertion query over HTTP will have to look like this:

my_house_data,SOURCE=arduino_1,PLACE=kitchen TEMP=20,HUMIDITY=74.3,LIGHT=788.88

And of course, the tags will be likely to be hard coded in each of your Arduinos (or any other microcontroller) but the values should be queried from the sensors dynamically. If you are asking what is the significant of measurement name, then my answer is (I can not get a clear answer from InfluxDB’s documentation) perhaps you would like to separate your data by month name, so the measurement for month June would be my_house_data_june, and when next month comes, you will update that code 😉

How to use create a database inInfluxDB

InfluxDB is in some senses similar to your average database system such as MySQL, as it has a kind of database->table->column structure. So in order to insert some data in InfluxDB, you need to have a database created before attempting to insert something into it. This can be done in many ways, but the easiest is to use the administration panel in the browser as below:

Query for creating a database in influxdb

The used query in this case is

CREATE DATABASE embedonix

Now you can start inserting data in the newly created database (in this tutorial we assume the name is embedonix).

How to use INSERT queries over HTTP in InfluxDB

The Line Protocol for HTTP uses POST request to the write script on the server which is hosting InfluxDB, followed by the name of the database that data should be inserted in. A correctly formed HTTP header then will look like this:

POST /write?db=embedonix HTTP/1.1");
Host: www.embedonix.com
User-Agent: Arduino/1.0
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 79

my_house_data,SOURCE=arduino_1,PLACE=kitchen TEMP=20,HUMIDITY=74.3,LIGHT=788.88

As you can see above, the POST is requested to the write script with the db argument set to the name of the database we want to put data in (here, ‘/write?db=embedonix’). After the Content-Length there is a blank line that tells the HTTP parser that the actual content starts here. The Content-Length is important to be exactly as the number of characters in the data, otherwise, the HTTP parser will refuse to handle the request!

In the next section, we will see how we can write data in InfluxDB using Arduino.

[nextpage title=”Sending data to InfluxDB using Arduino over HTTP”]We are going to use an Arduino (any model) with an Ethernet shield. You can also perform the same task with WiFi shield with some minor changes in the code.

Arduino Mega with Ethernet shield

The code

Since I only have 1 Arduino at hand and no actual sensors, I am going to emulate the test scenario in the code. But You can of course change to code according to your needs. The important part of the code is how to write into the buffer and how to calculate exact length of the content so we can make the HTTP parser of InfluxDB happy! This can be done through the magic of sprintf function which can write into a buffer char array and return number of written bytes.

Here is the code, I have comment it throughly so it should be obvious what it does. If you have any questions, feel free to ask in the comment section or in the embedonix’s forum.

#include <Ethernet.h>

/** Ethernet client instance */
EthernetClient client;

/** Default local IP address, in case of DHCP IP assignement fails */
const IPAddress eth_default_ip( 192,  168,  0,  99);

/** MAC Address of the ethernet shield */ 
byte eth_mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0xCD, 0xB8 };

/** Server address wich runs InfluxDB */
const byte eth_server[] = {192, 168, 0, 1};

/** InfluxDB HTTP port */
const int eth_port = 8086;


/** Size of the buffer for HTTP content */
const int bufferSize = 2048;

/** An character array filled with null terminate chars */
char buf[bufferSize] = {'\0'};

//some variables for generating places
const char kitchen[] = "kitchen";
const char garage[] = "garage";
const char living_room[] = "living_room";
const char basement[] = "basement";
int lastGivenPlace = 0;

/**
 * @brief: Returns a place out of 4 places we have!
 */
const char* get_place() {
  switch(lastGivenPlace) {
    case 0:
    lastGivenPlace = 1;
    return &kitchen[0];

    case 1:
    lastGivenPlace = 2;
    return &garage[0];

    case 2:
    lastGivenPlace = 3;
    return &living_room[0];

    case 3:
    lastGivenPlace = 0;
    return &basement[0];    
  }

  return "unknown_place";
}

/**
 * @brief Starts the ethernet shield as client
 */
bool eth_start(){
   Ethernet.begin(eth_mac, eth_default_ip);
   delay(2000); //delay to allow connection to be done

   //do a fast test if we can connect to server
   int conState = client.connect(eth_server, eth_port);

   if(conState > 0) {
    Serial.println("Connected to InfluxDB server");
    client.stop();
    return true;
   }

  //print the error number and return false
  Serial.print("Could not connect to InfluxDB Server, Error #");
  Serial.println(conState);
  return false;
}

/**
 * @brief Send  HTTP data to InfluxDB
 * @param data  Pointer to the beginning of the buffer
 * @param dataSize  Number of valid characters to send
 */
void eth_send_data(char* data, int dataSize) {
  //first we need to connect to InfluxDB server   
  int conState = client.connect(eth_server, eth_port);

  if(conState <= 0) { //check if connection to server is stablished
    Serial.print("Could not connect to InfluxDB Server, Error #");
    Serial.println(conState);
    return;
  }
  
  //Send HTTP header and buffer
  client.println("POST /write?db=embedonix HTTP/1.1");
  client.println("Host: www.embedonix.com");
  client.println("User-Agent: Arduino/1.0");
  client.println("Connection: close");
  client.println("Content-Type: application/x-www-form-urlencoded");
  client.print("Content-Length: ");
  client.println(dataSize);
  client.println();
  client.println(data);

  delay(50); //wait for server to process data

  //Now we read what server has replied and then we close the connection
  Serial.println("Reply from InfluxDB");
  while(client.available()) { //receive char
    Serial.print((char)client.read());
  }  
  Serial.println(); //empty line

  client.stop();  
}

/**
 * @brief Setups the peripherals
 */
void setup() {
  //Serial interface for debugging purposes
  Serial.begin(115200);
  delay(1000);
  eth_start(); 
}



/**
 * @brief Main loop
 */
void loop() {

  //A loop to send some data
  for(int i = 30; i < 50  ; i++) {
    
    //this will be number of chars written to buffer, return value of sprintf
    int numChars = 0;
    
    //First of all we need to add the name of measurement to beginning of the buffer
    numChars = sprintf(buf, "my_house_data,");
    
    //tag should have an space at the end 
    numChars += sprintf(&buf[numChars], "SOURCE=arduino_1,PLACE=%s ", get_place());
    
    //after tags, comes the values!
    numChars += sprintf(&buf[numChars], "TEMP=%d,", i);
    numChars += sprintf(&buf[numChars], "HUMIDITY=%.2f,", i * 1.03);
    numChars += sprintf(&buf[numChars], "LIGHT=%.2f", i * 10.5);

    //Print the buffer on the serial line to see how it looks
    Serial.print("Sending following dataset to InfluxDB: ");
    Serial.println(buf);

    //send to InfluxDB
    eth_send_data(buf, numChars);

    //we have to reset the buffer at the end of loop
    memset(buf, '\0', bufferSize);
    delay(1000); //some small delay!
  }
  
}

Notes regarding the code

Although I have commented the code thoroughly, here is a brief description of the code:

• Lines 4 to 16 are variables required for setting up and connecting to the server using the Ethernet shield. In my case, the Arduino was connected to my own laptop in a loop-back configuration, so the server address for InfluxDB would be obviously 192.168.0.1. If you are planning to connect your Arduino to a router or you are using WiFi, you must make sure to put the correct address in the eth_server[] variable!
• In line 20 we have defined an integer equal to 2048, which is the size of our buffer to write the insertion query in it.
• Line 23 is the definition and initialization of the buffer.
• the get_place function is just used to get an imaginary place out of 4 places
• the eth_send_data function is used to generate and send the correct HTTP POST request to the InfluxDB server
• the loop function is used to continuously send some values in the range of 30 to 50 as temperature and a multiple of it as humidity and light values

So all and all, what you need to pay attention is the correct usage of writing a message into a character array buffer using sprintf function and using its return values to know where to continue inserting into the buffer.

In the next step, we will see how does the data is being transmitted and how to see the raw data in the InfluxDB’s administration panel.[nextpage title=”Viewing the raw data”]After uploading the sketch in your Arduino, and making sure that it can connect to the computer which hosts the InfluxDB server, you should see something like the animation below, which are the Arduino’s serial port monitor and InfluxDB terminal:

If you are seeing something similar to the above animation, then everything is right! Otherwise, discuss in the comment section or in the forum and I will try to help you out.

To access the administration panel of InfluxDB, open a browser on the host computer and go to ‘localhost:8083‘. On the top of the page, select the database to work on (remember we named it embedonix if you have changed that name, then it is something else :P). After that in the Query text box, enter the following query:

SELECT * FROM my_house_data

Then you should see an overview of recently received data.

So far, these are raw data in the database. They can be viewed, queried and sorted based on metrics such as time, maximum and minimum and average. But This is not what we want! we want a nice and clean graphical presentation, which leads us to the next section, using Grafana![nextpage title=”Setting up Grafana to view InfluxDB data”]Grafana is most commonly used for visualizing time series data for Internet infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control. Grafana features pluggable panels and data sources allowing easy extensibility and a variety of panels, including fully featured graph panels with rich visualization options. There is built-in support for many of the most popular time series data sources. An example of data visualization by Grafana can be seen in the below image:

Grafana in action!

Installing and setting up Grafana on Linux

Just like InfluxDB, you can find the Grafana package for your Linux distribution (or Windows as well) in the download page on their website. Since we are working on Ubuntu, we need to download the .deb package. After you have downloaded it, install it using ‘dpkg -i [filename.deb]’ command. After installation, since Grafana registers itself as a service, you will need to start the service:

$wget https://grafanarel.s3.amazonaws.com/builds/grafana_4.0.2-1481203731_amd64.deb
$sudo dpkg -i grafana_4.0.2-1481203731_amd64.deb
$sudo service grafana-server start

After starting the Grafana server service, assuming InfluxDB is also running, head on to your browser and open the localhost at port 3000 which is the default port for Grafana (localhost:3000). There, the first step is to add a Datasource. Using the menu on top-left of the page, go to Data sources and add a new data source. Here you need to give your new data source a name and make sure to select the Type as InfluxDB. The address will be HTTP address of InfluxDB (http://localhost:8086). On the bottom of the page, enter the name of your database (remember we created a database named embedonix). After saving the data source, it is time to create a dashboard to get visualized data for our test scenario.

Head to Dashboards and select new to create a new dashboard. In the dashboard window, click on the Graph to create a new graph. Doing so will create a new panel named Panel Title which is filled with some default data. If you right click on the title you will be able to select Edit option to edit the query. What we want is a query to get data from our my_house_data measurements. Since it is hard to explain this process with words, please watch the quick video below to see how to configure Grafana to show the saved values in InfluxDB!

With a bit of practice, you can figure out how to make a great dashboard! Just give it a try 😉