Reproducible developer setup for RIOT

Development
#1

Hey there everyone,

I am looking for some feedback on an idea for a problem I have noticed with newcomers to RIOT.

I have found that students, particularly those inexperienced with Linux, can spend the better part of a day (sometimes days) just getting an environment setup to build RIOT and flash a board. There are different ways of doing this, many that are outdated or unmaintained.

I would say the current “recommended system” is defined by our docker setup. This is missing the flashing and debug setup however…

I propose the following:

  • Add the flasher + debug capabilities for to riots docker file with the help of the --device flag and possibly some discovery scripts
  • Add some docs and basic scripts to help with setup of windows and mac systems so they can run RIOT via a docker entrypoint
  • Add github actions to test that these are keeping up-to-date
  • Deprecate outdated setup scripts/boxes so there is one clear entrypoint
  • Make a machine readable make print-versions to allow comparing the output across different platforms (ie, docker running in windows vs docker running in mac vs docker running in ubuntu vs native developer machine)
  • Add clear documentation to say this is the recommended system for development
  • Add a dev container for those vs-code users out there

If we have this we solve the problem of developing with other operating systems and distributions and can better define what is needed to run RIOT.

The one issue that I haven’t included is setting up on a native ubuntu box and developing without docker. For this I would imagine either a setup_script.sh or and ansible playbook or something. The problem would be that we now need to maintain both the Dockerfile and the script. If we use the make print-versions we can force them to be in sync with each-other but it adds some overhead.

Anyways, please let me know if people think this is actually an issue in RIOT, if the proposed solution seems reasonable, and if there are any corner-cases I am missing.

Thanks!

3 Likes
#2

Seems related to Setup of development environment - #6 by oleg

#3

Hey Kevin,

thanks for pushing this topic. Are you considering adding this to (or updating) Getting started?

#4

Yes but I am under the impression if there are not tests enforcing something it will not be maintained. Only updating the docs will solve the problem for now but may quickly become outdated. This is why I want to add some automated setups and method for evaluating the setups are correct.

The docs will come once we have that in place.

#5

You are totally right here. I find projects that are overly reliant upon docker to become unmaintainable and very hostile to people trying to learn to do more. Too much situational knowledge is encoded in the Dockerbuild file rather than clarified or simplified. (I found this with the ESP32 builds, btw)

I think that we pander too much to students (and private sector developers) who only know windows. Computers are cheap, VMs are cheaper. Learning *nix is not that big a hurdle.

#6

Note that Windoofs these days ships with a Linux kernel. I think just searching for “Ubuntu” in the Microsoft app store and a click on install will provide you with a working Ubuntu shell, from where installing python3, wget, unzip, newlib and gcc is the only thing missing to be able to compile without docker.

This Windoofs Subsystem for Linux is really awesome. It is the first time that an environment for productive development is in reach of Windoofs users. And after a few weeks of time some realize that they actually can get the Linux experience without having to boot Windoofs first. And that the native Linux experience is so much faster and leaner than WSL. And that you don’t end up with some Xbox Foo and Weather Forecast Bar installed against your will every know an then. And that your default browser choice is respected and not questioned or silently reverted after every second OS update that fixes critical security bugs.

1 Like
#7

Is that a general statement or RIOT specific? General, I agree, RIOT, I would say the fact that our Vagrant box setup is still referencing Ubuntu 16.04 says otherwise.

#8

Hey Marian,

On Mon, Dec 12, 2022 at 02:45:43PM +0000, Marian via RIOT wrote:

Note that Windoofs these days ships with a Linux kernel. I think just searching for “Ubuntu” in the Microsoft app store and a click on install will provide you with a working Ubuntu shell, from where installing python3, wget, unzip, newlib and gcc is the only thing missing to be able to compile without docker.

do you know if there’s actually a way to flash a device from WSL(2)? So far, my understanding was that native and building for real hardware will work but there’s no access to USB.

Cheers Oleg

die_if_kernel(“Whee… Hello Mr. Penguin”, current->tss.kregs); linux-2.2.16/arch/sparc/kernel/traps.c

#9

This Windoofs Subsystem for Linux is really awesome.

I have failed the last time I tried to use WSL2 to flash and debug a platform but I hear the they finally fixed that so maybe that would do (for Windows).

If the general consensus is to say try not to do everything in docker then and use VM/WSL2/anything then we still lack a nice setup script that is maintained… and I ask, ansible or basic script or something? If we have that how to we keep it up-to-date, should we still have a method of comparing a developer setup to the CI accepted image? Should we invest time in automating the checks (github action spin up and instance in the different VMs)? Finally, should this update script be able to re-run (ie, it shouldn’t just work on an empty image)?

1 Like
#10

Not exactly correct, the Vagrant setup uses 18.04. I agree that it should be upgraded.

#11

You can call Windows binaries from withing WSL2. So if you add a symlink to the Windows version of JLinkExe or edbg.exe in place where the RIOT build system would expect the native version, you can use that to flash from within WSL2.

This only works because the Windows version and the Linux version of those tools expect the same parameters. AFAIK this does not work with OpenOCD.

1 Like
#12

RIOT/README.md at master · RIOT-OS/RIOT · GitHub says 16.04, another point that docs can too easily become out-of-sync.

#13

The README update was indeed missed in the original PR. Should be an easy documentation fix :slight_smile:

#14

A bit of both.

That we still use Ubuntu 16.04 in vagrant is a sign that our setup is old, and not really reproduceable. I love dockerbuild. I really do. It’s a super sharp sword that samurai should use to deploy reproduceable setups into production (yes, like CI). But, it’s too complex for newbies, and it shields them from ever learning anything. Learning Dockerbuild in order to figure stuff out is too high a hill to climb. Of course, people say, but they aren’t supposed to learn that… but I claim that means that they can never learn anything. And mid-level people can’t learn it either.

#15

Yes it is, but not directly. The Linux kernel in Windoofs has no USB access at all (despite the Windoofs users spamming all forums with claims that USB is in WSL fully supported and just works - they seem to not understand the difference between USB storage being accessible and USB access). But if you launch an application from Linux within WSL and it is not in $PATH, it will forward that to Windoofs which will then search in %PATH% and run the Windoofs app. So running explorer.exe . in Linux will actually result in a Windoofs Explorer window popping up that is navigated to the path the Linux shell is currently in. That integration really works like a charm.

I think that just adding a symlink to OpenOCD.exe in /usr/local/bin/openocd may just work. At least make BOARD=<board> OPENOCD=OpenOCD.exe flash will work, if OpenOCD.exe is found in Windoof’s %PATH%.

It may be either to go the J-Link route for Windoofs users though, I guess.

Similar, make term will work if the terminal is a Windoofs app and PORT is set to something that Windoofs app will understand. It will however launch a new window (even if it is a cli tool, I exepect a cmd.exe window jumping up).

#16

As much as I like to agree with you on the Linux part. Reality is, that’s not where a lot of people are coming from. And the question here is: Do we want people to use RIOT? Or are we fine with losing people, because it’s mostly Linux + RIOT? Regarding the VM topic btw. We are all here in industralized countries. The interesting markets where IoT will play a big role don’t have the same powerful machines as we have. They even rock old Intel 4 or Core2Dous. Some even use Smartphones for development. I even gave it a try. It really works. You can develop decently well on Smartphones ^^

Regarding Windows. I tried WSL2 one time and couldn’t flash the MCU. Is that something that works nowadays under Windows with docker?

Well, it does work when you do these USBoverIP shenanigans. Never tried it though. Btw, couldn’t you just take the binary and flash it from Windows, not WSL2? At the end of the day you just have to make it easy to use for the user. It can be ugly behind a fancy front. Speaking of fancy front: Windows users and Mac user need an easy installer. That’s just how it is. At the end of the day it has to be the same experience as using PlatformIO in an IDE or the Arduino IDE. That’s the level of convience you have to beat in order to get these type of user to RIOT. So an integration in IDEs is probably also needed. I used my clout on twitter in order to push for a platformIO integration. Nothing has happened so far. Maybe that’s the point to pick that up, talk to them and help them integrate it. Aka adding the missing pieces.

#17

Right, which run a Linux desktop without a problem, but won’t boot Windows 10, and will never, ever run Windows 11?

#18

That’s true. They either run a Linux or a stone old Windows. That’s indeed an interesting question. I can try find out more about that. Especially the Windows 11 part. How about just make the user experience great for Windows, Linux and MacOS? I mean, it’s not even that user-friendly on Linux. But then we have to go into the whole shitshow of package managment under Linux.

#19

It is. As Benjamin pointed out. You can symlinks the Windows binaries to Linux to mimic the behavior within WSL2. Which means, install JLink, openOCD or EDBG Windows binaries, symlinks them to WSL2 (/usr/local/bin, or anywhere…) and it just works. No needed for USB over IP this way since we’re using Windows binaries from WSL2.

1 Like
#20

I still use my wierd shell script to do setup from scratch on Ubuntu - seems to work OK for our preferred samr devices… myRIOT/INSTALLRIOTonUBUNTU at master · kmartinez/myRIOT · GitHub cheers Kirk