RIOT OS for data logger system with focus on maintenance

Dear RIOT OS community,

I’m Daniel a PhD in Hydrology at GFZ Potsdam and currently working on a data logger system for use in extensive field campaigns. As far as I have understood, RIOT OS seems to be very suitable for the tasks I have in mind. Since I am new to using an OS on microcontrollers (rather than running simple single-thread applications), I would very much like to discuss with a RIOT expert (or several) what is possible and what is an illusion of my perception. Here are some points to give you a bit of an idea what I mean:

  • Adding pre-compiled sensor driver packages to a running application Let’s say we have 20 stations running but with varying numbers and types of sensors (e.g. sensors monitoring trees only in forests, discharge sensors only at rivers, etc.). Is it possible to have a common application running on all stations and adding/removing pre-compiled sensor driver packages to change sensor setup? It would allow easier on-site maintenance, where fast adding/removing sensors would not lead to changing source code and compiling a new application for one particular station (maintenance of course always happens in winter, at night, with strong winds, heavy rain, in a remote forest, hard to work on source code :slight_smile: ).

  • Having a web interface running, accessible via Ethernet or BLE (for on-site maintenance) and remote (e.g. via LTE CAT M1/NB1 for administration) to allow configuring sensor setup (number and type) as well as general configuration (measurement interval, time of data upload, etc.) A simple graphical web interface would ease the maintenance for the technician as he would “only” need to change configurations rather than changing source code and compiling (see point above). Since I am working currently with OpenMediaVault, I really appreciate their web interface concept, with the possibility to revert all currently made changes.

  • All these configurations would need to be stored locally, I imagine as JSON on an EEPROM with some kind of file system. Is this how it is done in general?

  • All configurations should also be synced with the remote server (e.g. running ODM2)

I hope these points make some sense even if the answer is: No, what you want is not possible. And if not, I am happy to learn more. As mentioned before, I would appreciate a discussion about this use case and the potential use of RIOT OS with experienced people who find it interesting. In case spamming the user list is not desired you can of course contact me via my email address

Enjoy your weekend and stay safe Daniel

I find it interesting enough to keep this discussion on the open list.

  * Adding pre-compiled sensor driver packages to a running application Let's say we have 20 stations running but with varying numbers and types of sensors (e.g. sensors monitoring trees only in forests, discharge sensors only at rivers, etc.). Is it possible to have a common application running on all stations and adding/removing     pre-compiled sensor driver packages to change sensor setup? It would allow easier on-site maintenance, where fast adding/removing sensors would not lead to changing source code and compiling a new     application for one particular station (maintenance of course always happens in winter, at night, with strong winds, heavy rain, in a remote forest, hard to work on source code :slight_smile: ).

An easier solution would be to just compile in all your sensor drivers into one image. Usually drivers will have a way to detect if a sensor is connected (and if not, it should be added).

When you access sensors via SAUL you give it a sensor type and if a matching sensor present and initialized correctly, it will be used to get you the data.

You can of course do OTA updates of the whole firmware using SUIT.

  * Having a web interface running, accessible via Ethernet or BLE (for on-site maintenance) and remote (e.g. via LTE CAT M1/NB1 for     administration) to allow configuring sensor setup (number and type) as well as general configuration (measurement interval, time of data upload, etc.)     A simple graphical web interface would ease the maintenance for the technician as he would "only" need to change configurations rather     than changing source code and compiling (see point above). Since I     am working currently with OpenMediaVault, I really appreciate their web interface concept, with the possibility to revert all currently made changes.

Consider using an App that runs on a phone and talks to the sensor using Bluetooth Low Energy (BLE). If you want to use IP, we have a CoAP server in RIOT (although it should be possible to run a HTTP server using LwIP) and there are CoAP <-> HTTP bridges.

Again consider running most of the logic on the client.

  * All these configurations would need to be stored locally, I imagine as JSON on an EEPROM with some kind of file system. Is this how it is done in general?

Unless you want to store the data on an SD card for easy reading/editing on a PC, you'd probably much rather use a binary protocol like ProtoBuf or CBOR. And since wear-leveling flash file systems like littlefs2 and spiffs exist, you can also just use Flash memory instead of an EEPROM.

  * All configurations should also be synced with the remote server     (e.g. running ODM2)

Mind you that the nodes won't do TCP. You can run a CoAP server and have the nodes connect to that to upload data / check for new configurations. What you then use in the backend is up to you.

Best, Benjamin

* Having a web interface running, accessible via Ethernet or BLE (for on-site maintenance) and remote (e.g. via LTE CAT M1/NB1 for administration) to allow configuring sensor setup (number and type) as well as general configuration (measurement interval, time of data upload, etc.) A simple graphical web interface would ease the maintenance for the technician as he would "only" need to change configurations rather than changing source code and compiling (see point above). Since I am working currently with OpenMediaVault, I really appreciate their web interface concept, with the possibility to revert all currently made changes.

Okay, that point is a bit tricky. At least tricky, if you want to have this web interface directly on the Microcontroller. I would use CoAP for this use-case. It makes the most sense. But there is one issue: Normal browser, like Mozilla, Chrome etc. don't support CoAP by default. There is Copper(https://github.com/mkovatsc/Copper), but as you can see it is discontinued. You can use an old browser, but I guess your point was to use a modern browser, right? There are other options as well, of course. LibCoAP is well supported. So, you can use it in a programming language of your choice. Every major language has some kind of CoAP lib. So, you could provide an CoAP interface you can interact with. Writing a CLI for it, shouldn't be a big deal. From the remote point of view: Just host a central management frontend which interacts with the devices. Collecting configs, collecting data etc. You probably also have to use edge device/border router/gateway in your setup. So, you could also put an instance of this web interface there as well. So that you have a web interface on-site as well. But having a CLI backup is probably also a good idea in case the web interface crashes. You can also have it more decentralized and let the edge device collect the data in some way. There is room for different architecturial decisions.

*All these configurations would need to be stored locally, I imagine as JSON on an EEPROM with some kind of file system. Is this how it is done in general?

Not sure, if JSONs is the best idea here, since you have to parse the JSON on the MCU. Using a filesystem in RIOT is quite simple. So, you can store it in files. When you call the API, you change the value in memory and on the file system. When the device wakes up it loads the configuration from the filesystem. I would also like to hear the input from other people, since I didn't had dynamic configuration yet.

*All configurations should also be synced with the remote server (e.g. running ODM2)

Just provide an API for this at your devices. So that you can change the configuration remotly. Of course, this should be done in a secured way.

Maybe it makes sense to use Web of Things (https://www.w3.org/WoT/) in your use-case. Especially considering you want to collect all these data, change the configuration etc. There is the WoT Scripting API which makes it quite simple to interact with the device and write a web frontend for it, once the WoT TD is present. Node-WoT(https://github.com/eclipse/thingweb.node-wot) is quite an advanced implementation of it. I wrote a PoC implementation for RIOT (https://github.com/w3c/wot-testing/tree/master/events/2020.09.Online/prototypes/RIOT-OS). If that sounds interesting to you, just ping me. I could imagine that you write some generic plugin/extension for ODM2 and support WoT TD on it. So that you can provide an interoperability layer. Eventually you could also replace your device with commercial available ones. They just have to support the WoT TD.

best regards Philipp

Hi Daniel,

see inline some answers and ideas for your use case.

Kind regards, Marian

Dear RIOT OS community,

I'm Daniel a PhD in Hydrology at GFZ Potsdam and currently working on a data logger system for use in extensive field campaigns. As far as I have understood, RIOT OS seems to be very suitable for the tasks I have in mind. Since I am new to using an OS on microcontrollers (rather than running simple single-thread applications), I would very much like to discuss with a RIOT expert (or several) what is possible and what is an illusion of my perception. Here are some points to give you a bit of an idea what I mean:

  * Adding pre-compiled sensor driver packages to a running application     Let's say we have 20 stations running but with varying numbers and     types of sensors (e.g. sensors monitoring trees only in forests,     discharge sensors only at rivers, etc.). Is it possible to have a     common application running on all stations and adding/removing     pre-compiled sensor driver packages to change sensor setup? It would     allow easier on-site maintenance, where fast adding/removing sensors     would not lead to changing source code and compiling a new     application for one particular station (maintenance of course always     happens in winter, at night, with strong winds, heavy rain, in a     remote forest, hard to work on source code :slight_smile: ).

It would certainly be possible to use a common firmware that targets the "full-fledged" node, but flash it on nodes with fewer sensors plugged in. Our goal is that drivers perform during initialization a self check (usually read out manufacturer and device ID and compare it against the expected values) - if no such sensor is present, the initialization should return with an error code. So my proposal is to just let the firmware detect upon start up which sensors are present. Sadly, you will likely not be able to use auto initialization with that approach, as you really want to check the return value of the initialization to verify if the sensor is available and functional. (Well, you could use SAUL with auto-initialization, as only successfully initialized sensors should be registered in the SAUL registry.)

Note however, that it is pretty much impossible to detect hardware configuration at runtime due to the plethora of different and incompatible communication standards and the lack of any "plug and pray" protocols. So my suggestion would only work if for each sensor the configuration is known at compile time and each sensor can either be present in this very configuration, or being unconnected.

But I fear that the test coverage for this scenario is not too well, so be prepared to file a few bug reports regarding the self checks. (I expect some drivers to still initialize even without hardware present, and some other drivers to infinitely loop while trying to communicate with non-existing hardware.)

  * Having a web interface running, accessible via Ethernet or BLE (for     on-site maintenance) and remote (e.g. via LTE CAT M1/NB1 for     administration) to allow configuring sensor setup (number and type)     as well as general configuration (measurement interval, time of data     upload, etc.)     A simple graphical web interface would ease the maintenance for the     technician as he would "only" need to change configurations rather     than changing source code and compiling (see point above). Since I     am working currently with OpenMediaVault, I really appreciate their     web interface concept, with the possibility to revert all currently     made changes.

That would certainly be possible to implement on top of RIOT. I'm unaware of anyone running an HTTP server on RIOT, but that shouldn't be too difficult to achieve.

However, maybe a CoAP server would be a better option? By reusing the core concepts of HTTP based REST APIs (same message-response model, URIs, content formats (lightweight MIME Types), ...) CoAP can be trivially reached from HTTP by using cross protocol proxies that transparently translate between HTTP and CoAP. And you might be able to use the same CoAP interfaces for both human 2 machine maintenance scenarios as well regular m2m communication.

Btw: lwm2m could also be something you might want to check out as an option for device management.

  * All these configurations would need to be stored locally, I imagine     as JSON on an EEPROM with some kind of file system. Is this how it     is done in general?

Usually the overhead of file systems is considered to be too huge for EEPROM devices. RIOT has the EEPROM registry module that works well as a low-overhead key value storage - which could be viewed as a primitive non-hierarchical file system. That should actually be quite a good match for storing configuration data.

If you want to go with an actual file system, you're likely better off with flash memory or SD cards as storage.

You might want to look at CBOR as alternative to JSON. It uses pretty much the same data model as JSON, but is more efficient in terms of storage and in terms of parsing overhead. Same as CoAP and HTTP, you can just convert data between the formats (with some limitations), but generally CBOR should be better suited for constrained devices.

  * All configurations should also be synced with the remote server     (e.g. running ODM2)

I hope these points make some sense even if the answer is: No, what you want is not possible. And if not, I am happy to learn more. As mentioned before, I would appreciate a discussion about this use case and the potential use of RIOT OS with experienced people who find it interesting. In case spamming the user list is not desired you can of course contact me via my email address daniel.beiter@gmx.at

Actually, I personally would like to hear more about how people are using RIOT or intending to use RIOT. Have this context information can be quite beneficial for making design decisions in RIOT - and hearing some success stories are quite motivating for the RIOT developers. So from my point of view you are very welcome to keep this discussion here.

I'm Daniel a PhD in Hydrology at GFZ Potsdam and currently working on a

    > data logger system for use in extensive field campaigns. As far as I     > have understood, RIOT OS seems to be very suitable for the tasks I have     > in mind. Since I am new to using an OS on microcontrollers (rather than     > running simple single-thread applications), I would very much like to     > discuss with a RIOT expert (or several) what is possible and what is an     > illusion of my perception. Here are some points to give you a bit of an     > idea what I mean:

    > * Adding pre-compiled sensor driver packages to a running application     > Let's say we have 20 stations running but with varying numbers and     > types of sensors (e.g. sensors monitoring trees only in forests,     > discharge sensors only at rivers, etc.). Is it possible to have a     > common application running on all stations and adding/removing     > pre-compiled sensor driver packages to change sensor setup? It would     > allow easier on-site maintenance, where fast adding/removing sensors     > would not lead to changing source code and compiling a new     > application for one particular station (maintenance of course always     > happens in winter, at night, with strong winds, heavy rain, in a     > remote forest, hard to work on source code :slight_smile: ).

Yes, this is certainly possible. I don't think you'd want to load binary packages, but rather just compile it all in, and the configure which GPIO should do what and mean what.

    > * Having a web interface running, accessible via Ethernet or BLE (for     > on-site maintenance) and remote (e.g. via LTE CAT M1/NB1 for     > administration) to allow configuring sensor setup (number and type)     > as well as general configuration (measurement interval, time of data     > upload, etc.)

This is possible if you put the right hardware in. Mesh IPv6 via RFC6550 (RPL) is common. I've run web interfaces like this, including one case where it was SLIP over a bit-bashed GPIO [that was Contiki on an MSP430, but...]. A one page app with an API is probably best.