On prototyping an async Rust runtime for RIOT

Development
1

Thanks @Kaspar @emmanuelsearch for sharing the latest overview/details on Rust integration efforts for RIOT:

Because we, IETF ANIMA Minerva Project (@mcr @j-devel), have also been working on our custom “cross-board oriented” async Rust runtime for RIOT, we particularly got interested in “Prototype RIOT over async Rust framework” part, which is assuming the Embassy ecosystem [1].

Related to the topic, we would like to share some additional insights that we gained though implementing our version [2] of async Rust runtime for RIOT.

In our case, we didn’t start with the existing 3rd party Embedded Rust components (and having to grapple with complexity for adaptation/integration); rather, we took a minimalistic/bottom-up approach:

  • port the (bare-metal minimalistic) blog_os async Rust executor [3] to a no_std RIOT-compatible crate [4],
  • bind existing RIOT C APIs (e.g. timer, gcoap) by Rust-C-FFI,
  • adapt them as Rust Future/Stream, and
  • expose them as Rust async function API

what we got working (thus far)

  • single threaded async Rust runtime (akin to JavaScript runtime), capable of
    • spawning Rust Future<Output = ()> instances
    • spawning Rust Stream processors (useful for implementing e.g. Rust-based coap servers)
    • executing async versions of some RIOT Timer/Gcoap API, e.g.
      async fn async_sleep(msec: u32)
async fn async_set_timeout<F>(msec: u32, cb: F)
async fn async_gcoap_get(addr: &str, uri: &str) -> Vec<u8>

what we learned/proved

  • binding existing RIOT (networking) C APIs, and then
  • wrapping/adapting them as Rust Future/Stream works well, hence
  • CROSS-BOARD (xbd) async abstraction of RIOT API is highly feasible/practical (as opposed to Embassy going “board-specific”)
  • we are pushing/following this path to build Rust-based MCU client (esp32) to be demonstrated in IETF ANIMA BRSKI interop

async-runtime-riot
async-runtime-riot597×600 21.1 KB

[refs]

  1. GitHub - embassy-rs/embassy: Modern embedded framework, using Rust and async.
  2. https://github.com/AnimaGUS-minerva/studio/tree/main/examples/xbd-net/src
  3. GitHub - phil-opp/blog_os at post-12
  4. https://github.com/AnimaGUS-minerva/studio/tree/main/examples/xbd-net/src/blogos12
1 Like
2

Hello j-devel,

interesting work, thanks for sharing!

  • wrapping/adapting them as Rust Future/Stream works well, hence
  • CROSS-BOARD (xbd) async abstraction of RIOT API is highly feasible/practical (as opposed to Embassy going “board-specific”)

I just skimmed the code, but it looks like there are quite some heap allocations. E.g. in xbd_ztimer_set() (which is just for testing IIUC), or in Xbd::set_timeout(). We can’t have heap allocations on non-MMU systems. I’d assume that getting rid of those is actually the hard part … What do you think?

3

Interesting. I did grapple with integration for a long while, then took the minimalistic/bottom-up approach of starting with the existing 3rd party Embedded Rust components and drop most of RIOT-c for now. :slight_smile:

But looking at the approach you’re presenting here, I think it should be really simple to hook embassy’s executor(s) into RIOT-c without much integration hassle, especially if embassy-hal is not being used. Maybe worth exploring if you’re start hitting the limits of the blog_os executor.

4

Many thanks for feedback!

We can’t have heap allocations on non-MMU systems. I’d assume that getting rid of those is actually the hard part … What do you think?

I should have made clear that non-MMU systems are currently out of our scope. Our Rust code assumes heap allocations (via no_std with alloc feature enabled). no_std Rust without alloc in conjunction with RIOT is unexplored, and also non-trivial IMO.

5

But looking at the approach you’re presenting here, I think it should be really simple to hook embassy’s executor(s) into RIOT-c without much integration hassle, especially if embassy-hal is not being used. Maybe worth exploring if you’re start hitting the limits of the blog_os executor.

I appreciate your insights about embedding Rust-executor(s) into RIOT-c. As you suggest, we would like to consider Embassy executor(s) when overcoming limitations, especially those pointed out in the “possible-extensions” section.

6

That’s tricky because RIOT doesn’t support any systems with a MMU. :wink:

The issue is not that you can’t do heap allocations at all, they are well supported and even used in some rare cases in RIOT code. The issue is that you have to be careful with them because there is nothing that can safe you from heap fragmentation.

So dynamic heap allocations, especially frequent and small ones are a big issue. You can do heap allocations on init if you are expected to keep the memory for the entire run-time of the app (that’s what MTD does) or if you have multiple application states between which the memory is freed.

Doing many small/variable sized allocations / deallocations can lead to ‘holes’ in the memory where you can’t satisfy an allocation even if there is in theory still enough free heap memory.

7

So dynamic heap allocations, especially frequent and small ones are a big issue.

Thanks for explanation. I think, we’d better check/evaluate the underlying heap allocator regarding this heap fragmentation issue.

8

There’s no solution - all allocators suffer fragmentation (some more, some less).

1 Like
9

I wanted to post about a netdev talk this fall about Rust in the Linux Kernel. The email announcing it was not public in the end. Ah, I found the link: https://netdevconf.org/0x17/sessions/tutorial/rust-for-linux-networking-tutorial.html It’s a tutorial. I don’t know much about the Rust integration into the Linux kernel; the talks at LPC last September was in conflict; I should look it up. I think they did not use cargo, but I am not certain.

I wanted to remote attend all of last week, but due to reasons I didn’t make much of it. I did catch the last 5 minutes of @Kasper’s talk, and I hope to see the video posted.

This is my take on where to go with RIOT-OS + RUST.

  1. We won’t get there overnight.
  2. There is a very big win for having a RIOT crate that just contains all our C code with some reasonable default Kconfig. It’s okay if it uses 80% of the code space on the target platform. The target audience are (new) people writing main() in RUST. I think we had a talk about this from Japan in 2021’s conference. It doesn’t have to be for every platform, just ones that are actually available.
  3. There are many bits of code that would benefit from being written in Rust, even if they call C functions and are called by C-functions. They would ideally be no_std, but really it’s when you need to keep track of some memory that is allocated that RUST wins. The key here (IMHO) is to find a way to just plop .rs files into our tree and have them work.
  4. The work that @j-devel has done is probably more advanced than many are ready for, but OTH, many will get to wanting async runtime for applications very soon.
10

I think the word MMU has not translated well.

11

The issue is that the allocator can’t move allocated memory around to defragment things. This is also an issue with many *nix programs. Some systems (Java poorly in my opinion, Ruby much better, Emacs…) have garbage collection that can move allocated memory around. And it’s even possible to return entirely empty pages to the OS (even creating holes). None of this possible in RIOT-OS, and it doesn’t matter what kind of allocator one uses.

I haven’t had a chance to look at the code, but I suspect that it can be taught to allocate at init time, and then maintain it’s own heap for async operations.

12

Thanks for sharing this … should have read this earlier!

So dynamic heap allocations, especially frequent and small ones are a big issue.

From my own work on the same topic, the pain point is that some structures need to be initialized in place (VFS directory entries, sock_udp_t), and Rust has no easy way to run a constructor into a Pin<&mut MaybeUninit<...>>.

I’ve played around with this a bit on my own, and after an initial very cumbersome approach, I found that writing an async RIOT executor based on Embassy can be done in <50LoC total, and can be fully decoupled from the implementations of sockets and timers.

My current worst sore point is that I don’t see easy ways for using the mutex or msg synchronization primitives in an async way; did you have any success with that?

13

Sorry for a late comment, but

Not sure, our approach to async API functions is to wrap RIOT-c callbacks into a async Stream processing. Maybe, this approach is limited in that we assume binding RIOT-c methods is the way to go.

14

All works well as long as the C API gives you callbacks; mutex and msg just do not have callbacks, but directly block one thread on the very operation that is pending.

1 Like