We also should discuss robot framework [1] as suggested by ELL-i at the
workshop. I did not look into it too much, but from as much as I gathered
it might be helpful to automate a lot of already existing tests in our
`tests/` directory. It's also integratable into Jenkins if this is
important to you [2].
@Kasper: more options means a higher probability that we will find a day on which everyone can attend the meeting.However, I do understand that it is quite a hassle to fill out a form with that many options (I only realized that after sending
the mail).
@Martine: Good point. We should definitely discuss the robot framework as well.
it would probably better if we have this meeting sooner than
later. That is why I deleted the last three days from the dudle (26-/27-/28- th of November).
This has the nice side-effect that the form is now less of a hassle to fill out.
This is where we have existing code for Robot Framework. ELL-i is able to run unit tests
in real hardware and apply tests with a logic analyser.
I’d like that whatever test system it is, an individual developer could run it even offline.
For this goal, virtualization seems to be the easiest path.
RF is Python and its test cases are in wiki markdown format. This makes it easy to separate test spec from the testing framework itself allows nonprogrammers to participate in some of the generating of test cases. I’m hoping Asif can join the meeting as he is the key RF person for ELL-i and has written bidirectional Python<->C hooks for running unit tests against the ELL-i emulator. The emulator itself is emulating a MCU including register controlled peripherals so we have some interesting tests coming up. It could be used in Riot OS context as well.
I expect to have some real hardware in hand (CC2538 based boards of my own design, and possibly some MSP430 boards after that) in the coming weeks (months if I don’t get into gear). I’d be more than willing to contribute some whatever time is needed to running tests on real hardware if it’s helpful (assuming that I have the gear, my LA isn’t the greatest but that could change).
On a related note, how much (if any) of RIOT’s code base is covered by real unit tests? Has any thought been given to a TDD type of unit test battery? With the all the refactoring going on in the network stack this might be the perfect time to get serious about writing some tests.
Automated hardware based regression testing:
The automated hardware regression testing using a logic analyzer would certainly be interesting. However,
I think the first version of the automated hardware based test system will be much more simplistic (basically just boards
connected to an internet connected computer).
The main reason for this is that we (well I and Oleg) came to the conclusion that we cannot possible
operate an automatic hardware based test system for each and every board we support (as you know the number of
boards we support is rising steadily).
So we had the idea to make the system distributed and easy to setup (both hardware- and software-wise).
Ideally, such a system would then allow external entities (companies / universities etc.)
to setup their own test system for the hardware they care about (or happen to have). The test coordination / building
of binaries would then be done by a central server (which would be controlled by us).
Virtualization / Simulation / CI systems:
I think no matter which CI system we will pick we can put it into Docker or something similar and we will definitely
insure that it is hassle-free to use.
RF:
I have to admit that I hadn’t had the time to look into the specifics of your test system. Since, I really like the way
you can specify test cases in RF I would like to hear more about your approach. It would be cool if Asif could join
us and give us a quick technical run down of the system (BTW: It might not be that we will copy your testing approach
but I am definitely intrigued by RF).
@Adam:
Testing the network stack is one of the major concerns right now. AFAIK Martine has put quite some effort into making the
network stack testable (she also wrote a bunch of unit tests for network stack related modules).
Concerning the code coverage: I think its safe to say that not a lot of modules are covered by actual unit tests. Most tests
in the test directory are what I would call simple acceptance tests.
As part of our effort to improve the testing system we will definitely try to convert some of those tests either into more sophisticated
acceptance tests (using RF) or unit tests (or both if applicable / sensible).
Regarding your offer: First of all thank you for your offer. The way I see it there is currently no need for a rush concerning the hardware. We (well mostly I and maybe Martine) will need to develop / configure our new test
system first. So it could take a few weeks until we have something you
could install on a linux box.
I’d like to announce that the dedicated RIOT testing meeting will take place on:
Wednesday the 19th of November at 01:00 pm CET.
We will use PlaceCam for the meeting. If you are using Linux you need to install
PlaceCam beforehand. PlaceCam can be found at [1].
Please note that you don’t need to create an PlaceCam account.
Access to the conference call will be provided by a link which I will share on this
list shortly before the meeting starts.