Dedicated RIOT Testing Meeting

Dear RIOTers,

in order to discuss possible enhancements to our current test setup I would like to invite you to a dedicated RIOT testing meeting.

The main topics to discuss will be:

  • Automated hardware based regression testing.

  • CI systems: Jenkins / buildbot or both?

  • Simulations / Virtualization as part of our testing strategy.

  • Integration of additional static code analyzers (like coverity) into our CI system.

If you are interested, please take part in the dudle poll [1]. The meeting

will probably take place at FU Berlin but we could also organize it as a virtual meeting (similar to the virtual developer meeting) if necessary.

[1] https://dudle.inf.tu-dresden.de/testingtheriot/

Best,

Philipp

Hi,

We also should discuss robot framework [1] as suggested by ELL-i at the workshop. I did not look into it too much, but from as much as I gathered it might be helpful to automate a lot of already existing tests in our `tests/` directory. It's also integratable into Jenkins if this is important to you [2].

Cheers, Martine

[1] http://robotframework.org/ [2] https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin

Hi again,

Hi,

@Kasper: more options means a higher probability that we will find a day on which everyone can attend the meeting.However, I do understand that it is quite a hassle to fill out a form with that many options (I only realized that after sending the mail).

@Martine: Good point. We should definitely discuss the robot framework as well.

regards, Philipp

Hi,

it would probably better if we have this meeting sooner than later. That is why I deleted the last three days from the dudle (26-/27-/28- th of November). This has the nice side-effect that the form is now less of a hassle to fill out.

regards, Philipp

Hello,

You don't have to, but being a commercial and old-school developer I would definitely add:

  - Sell to customers. Let them test it

  - Make some useful applications with it and install it on those devices.

  - Just put it in production - sort out problems as they arise.

Regards

David

Something like this : ?

  - https://www.facebook.com/video.php?v=818668814853955&fref=nf

This is where we have existing code for Robot Framework. ELL-i is able to run unit tests in real hardware and apply tests with a logic analyser.

I’d like that whatever test system it is, an individual developer could run it even offline. For this goal, virtualization seems to be the easiest path.

RF is Python and its test cases are in wiki markdown format. This makes it easy to separate test spec from the testing framework itself allows nonprogrammers to participate in some of the generating of test cases. I’m hoping Asif can join the meeting as he is the key RF person for ELL-i and has written bidirectional Python<->C hooks for running unit tests against the ELL-i emulator. The emulator itself is emulating a MCU including register controlled peripherals so we have some interesting tests coming up. It could be used in Riot OS context as well.

  • t

I expect to have some real hardware in hand (CC2538 based boards of my own design, and possibly some MSP430 boards after that) in the coming weeks (months if I don’t get into gear). I’d be more than willing to contribute some whatever time is needed to running tests on real hardware if it’s helpful (assuming that I have the gear, my LA isn’t the greatest but that could change).

On a related note, how much (if any) of RIOT’s code base is covered by real unit tests? Has any thought been given to a TDD type of unit test battery? With the all the refactoring going on in the network stack this might be the perfect time to get serious about writing some tests.

Hi Teemu and Adam,

@Teemu:

  • Automated hardware based regression testing: The automated hardware regression testing using a logic analyzer would certainly be interesting. However, I think the first version of the automated hardware based test system will be much more simplistic (basically just boards connected to an internet connected computer). The main reason for this is that we (well I and Oleg) came to the conclusion that we cannot possible operate an automatic hardware based test system for each and every board we support (as you know the number of boards we support is rising steadily). So we had the idea to make the system distributed and easy to setup (both hardware- and software-wise). Ideally, such a system would then allow external entities (companies / universities etc.) to setup their own test system for the hardware they care about (or happen to have). The test coordination / building of binaries would then be done by a central server (which would be controlled by us).

  • Virtualization / Simulation / CI systems: I think no matter which CI system we will pick we can put it into Docker or something similar and we will definitely insure that it is hassle-free to use.

  • RF: I have to admit that I hadn’t had the time to look into the specifics of your test system. Since, I really like the way you can specify test cases in RF I would like to hear more about your approach. It would be cool if Asif could join us and give us a quick technical run down of the system (BTW: It might not be that we will copy your testing approach but I am definitely intrigued by RF).

@Adam: Testing the network stack is one of the major concerns right now. AFAIK Martine has put quite some effort into making the network stack testable (she also wrote a bunch of unit tests for network stack related modules). Concerning the code coverage: I think its safe to say that not a lot of modules are covered by actual unit tests. Most tests in the test directory are what I would call simple acceptance tests. As part of our effort to improve the testing system we will definitely try to convert some of those tests either into more sophisticated acceptance tests (using RF) or unit tests (or both if applicable / sensible). Regarding your offer: First of all thank you for your offer. The way I see it there is currently no need for a rush concerning the hardware. We (well mostly I and maybe Martine) will need to develop / configure our new test system first. So it could take a few weeks until we have something you could install on a linux box.

Best, Philipp

Hi everyone,

I’d like to announce that the dedicated RIOT testing meeting will take place on:

Wednesday the 19th of November at 01:00 pm CET.

We will use PlaceCam for the meeting. If you are using Linux you need to install PlaceCam beforehand. PlaceCam can be found at [1]. Please note that you don’t need to create an PlaceCam account. Access to the conference call will be provided by a link which I will share on this list shortly before the meeting starts.

[1] http://www.daviko.com/videokonferenz/3-1-Download.html

Best regards, Philipp

Here goes the PlaceCame link: http://placecam.de/call.php?c=VBYEpXi43MZ2MOrV~pegsMm6Z7woUbw.VRY0Qxal2pE-

Hi, Sorry I couldn’t attend. Will there be minutes?

Cheers, Martine

Yes, I will provide a link to the minutes soon.

Hi everyone,

the minutes of the meeting are now available at:

https://github.com/RIOT-OS/RIOT/wiki/Virtual-testing-meeting-on-November-19%2C-2014

Please feel free to comment and edit if you think I missed anything or something is unclear.

Best regards,

Philipp