Hi everyone,
as some of you may know, I'm currently writing my thesis on "designing a storage abstraction layer for the IoT" which includes writing an implementation of the concept for RIOT. Up until now, I've been working mostly on my own, and while I think I have a good understanding of what I'm trying to achieve, I don't have a lot of experience designing (storage) APIs for embedded devices. I thought that some of you could provide helpful input, or point me towards flaws/bad pracices in my concept that I'm not able to see. As such, I would like to present my train of thought for my concept as well as an outline of my current ideas regarding API design, and I would welcome your thoughts and input!
I'm not planning to go into much detail into how things will be implemented here, but please let me know if you have any questions or doubts about the feasibility/usefulness of anything!
The initial idea [^1] was to start from the POSIX file system API and go from there. A common file system API provides you with an unmanaged sequence of bytes (files), will let you create directories, control accessibility of a file depending on user privileges, create hard/soft links and more. My research suggests that a large portion of that functionality should not be the focus when implementing a data storage system for RIOT.
From the sensor network storage systems I've studied so far (Matchbox, ELF, Coffee/Antelope, Capsule), the consensus seems to be the focus on storing streams of immutable data (sensor values), possibly indexed by time.Mutability is also supported in more recent systems, but always under the assumptions that mutating already written data will consume significantly more memory due to the nature of flash memory (i.e. write once/block erase).
In my opinion, a solution that provides a higher-level abstraction compared to "files" should be developed for RIOT. My plan is to follow a similar approach to Capsule [^2] and implement an object-based storage systems. The general idea is that the system provides a number of different storage objects, e.g. an indexed stream, an array or a queue. For example (prefixed `wip` because it doesn't have a name yet):
wip_fd sensordata = wip_stream("sensor:humidity", sizeof(uint64_t));
wip_stream_append(&sensordata, 12345);
// Averaging a stream uint64_t sum = 0; wip_iter iter = wip_iterator(&sensordata); while(uint64_t value = WIP_NEXT(uint64_t, iter)) { sum += value; } ...
This would allow a more declarative usage of storage when developing an application for RIOT. Processing a large sequence of data without loading all of it into memory would be supported via iterators. My initial idea was to use callbacks and functions like "fold" and "foreach" instead, but this turned out to be pretty clumsy in C. Note that, while my concept does not involve the concept of "directories", one could implement storage object namespaces based a separator in the object name, e.g. `sensor` in the above example.
Swapping out large data structures also comes to mind (e.g. a RPL routing table in storing mode). ICNs may also profit from this. Storing mutable, unmanaged byte sequences (i.e. files) would also be supported.
wip_fd arr = wip_array("core:routingtable", sizeof(rt_entry));
rt_entry x = WIP_ARRAY_GET(arr, rt_entry, 42); x.metric += 1 WIP_ARRAY_SET(arr, x, 42);
I'm aware that it'd be possible to implement all this on top of the POSIX file system API, but I personally think that having a higher level of abstraction that covers the common use cases is preferable to placing the burden on the application developer. In any case, the storage system I'm envisioning will be detached from the lower level flash storage abstraction, so it'd be possible to develop other storage layers on top of it. I've already extracted Coffee (from Contiki) from their soruce tree [^3], but may not have the time to port it to RIOT.
Looking forward to your feedback!
Cheers, Lucas
[^1]: https://github.com/RIOT-OS/RIOT/pull/2474 [^2]: http://dl.acm.org/citation.cfm?id=1182827&CFID=566423331&CFTOKEN=15751545 [^3]: https://github.com/x3ro/coffee-fs