Process and Store Data

Store Data

In this section, we will demonstrate how the previously integrated MQTT data transfer is visualised within MQTT Explorer and how it communicates with the iot.hs-pforzheim.de server.

The MQTT data transfer functions by publishing sensor data from various IoT devices to a designated topic on the MQTT broker. In our configuration, the broker is the iot.hs-pforzheim.de server, which manages the reception, processing, and storage of the incoming data. This data is subsequently utilised by low-code workflow tools such as Node-RED or n8n for further processing before it is ultimately stored in a database, specifically InfluxDB in our setup. The university hosts both tools under https://iot.hs-pforzheim.de/nodered/ and https://iot.hs-pforzheim.de/n8n/, providing flexible options for designing the processing pipeline.

Example Overview

For our example we used the previously mentioned group0 configuration. The sensor configurations used for this setup are based on a polling period of 10000 milliseconds (10 seconds) for each sensor. The registered sensors are:

Humidity Sensor (humidity_v2_bricklet/VkX)
Temperature Sensor (temperature_v2_bricklet/Nz7)
Barometer (barometer_v2_bricklet/Hcx)

These configurations ensure that sensor data is regularly transmitted to the broker every 10 seconds, irrespective of changes in the data.

Example configuration for the Humidity Sensor

config_group0.json

"tinkerforge/request/humidity_v2_bricklet/VkX/set_humidity_callback_configuration": {
    "period": 10000,
    "value_has_to_change": false,
    "option": "off",
    "min": 0,
    "max": 0
},

This setup ensures that data is sent periodically even if the sensor readings have not changed.

Connecting to the Broker in MQTT Explorer

To visualise the incoming data, we use MQTT Explorer, a powerful tool that allows us to monitor MQTT messages in real-time. Follow the steps below to connect to the broker:

Open MQTT Explorer.
Add a new connection by clicking the “+” icon.
Enter the following details:
- Hostname: iot.hs-pforzheim.de
- Port: 8883
Connect to the broker.

Once connected, you will be able to view all topics being published under /tinkerforge/callback/#.

Monitoring Topics and Payloads

In the MQTT Explorer interface, you will see the hierarchical structure of topics and their associated payloads. The relevant data will appear under the /tinkerforge/callback/ topics, where individual sensors publish their values. Below are examples of the sensor data visualised in MQTT Explorer:

Humidity Sensor Data

The humidity_v2_bricklet/VkX sensor publishes humidity data as shown in the example above. In the Value section of MQTT Explorer, you can see a payload such as:

{
  "humidity": 9999
}

This payload indicates the current humidity reading from the sensor.

Temperature Sensor Data

The temperature_v2_bricklet/Nz7 sensor publishes temperature data. In MQTT Explorer, you will see the temperature value in the following format:

{
  "temperature": 2381
}

The number represents the temperature reading.

Barometer Sensor Data

The barometer_v2_bricklet/Hcx sensor sends air pressure and altitude data. As displayed in MQTT Explorer, the altitude value is continuously updated, and the history shows changes over time. For example:

{
  "altitude": 470639
}

The history of the values allows you to compare previous messages and track the changes.

Data Processing with Node-RED and n8n

Once the data is received via MQTT, it enters the processing pipeline in Node-RED or n8n, where the payload undergoes a series of operations before being dispatched to the storage layer. Within these low-code tools, the data can be filtered, transformed, or enriched according to specific conditions or rulesets tailored to the application’s requirements. For instance, you may apply threshold filtering, convert units, or aggregate multiple data streams into a single output. This flexibility makes Node-RED and n8n ideal platforms for real-time data manipulation. The precise steps and logic behind this processing will be elaborated in the subsequent sections.

Storing Data in InfluxDB

After processing, the data flows into InfluxDB, a time-series database known for its efficiency in handling high-volume data with temporal components. The data is stored in a designated Bucket, where each record is time-stamped, enabling powerful querying capabilities and facilitating real-time analytics. InfluxDB’s time-series nature allows for detailed tracking of sensor metrics, trend analysis, and anomaly detection. The process of configuring the data flow from Node-RED to InfluxDB, as well as the database’s storage architecture, will be discussed in greater detail in the upcoming sections.

MQTT Topics Introduction to InfluxDB