Hi,
Now that I'm happily running automated tests for my SAMD21 board(s) I am wondering which tests should succeed. There are several that fail, but I don't know if that "normal" or not.
Some examples of test that fail
xtimer_periodic_wakeup: hangs at the end, last couple of printf don't come out tests/thread_flood: [ERROR] expected 30, created 29
Is there maybe a wiki with the status of the tests (per board?).
Hi Kees,
To my knowledge, there is no wiki page for this kind of thing. The simplest is probably to open an issue and list there the failures with their output. This way we can easily track the on going work to fix them.
See [1] as an example.
Alex
Hi,
every failing test indicates a bug. (From time to time the bug is found in the
test rather than in the code to test.) Feel free to open an issue for the
failing tests. I personally favor a tracking issue that lists each failing test
with a checkbox. It is motivating to tick them of one by one; and with the
tiring task of bug hunting some motivation is always welcome 
Kind regards, Marian
OK, first one created. I'm afraid there are many more to come.
The reason I ask all these questions is that I’m adding support for
another SODAQ board. And I was trying to do it the right way and
run some tests. Maybe that was a mistake
Anyway, I’ve created the first issue.
Hi,
You can just put all of them in the same issue. It will be easier to track. That is what is done in [1].
Alex
Well, not easier for me, I'm afraid. It will delay adding my board
even longer (which is already several months in progress).
And it's just a hobby for me. Otherwise it begins to look like
work
Nevertheless, I'll see what I can do.
Hi,
Hi,
"... FAILED (due to $reason)", and maybe not change return code to something that's an error.
While I like this idea I foresee a usage problem since some tests in that script fail due to missing root permissions (or missing scapy_unroot). If we say 'FAILED (due to missing root permissions)' I virtually expect any sane developer to re-run the script then with `sudo`, leading to people wondering why they are not able to compile applications anymore, just using `make all`, as the `bin` directory was created with root permission. Sure this can be easily fixed, I just say "mind the consequences" ;-).
Best regards, Martine
I don't agree with the "sane" part in that sentence. 
Let me explain a little bit more why I am struggling right now.
First thing you should know is that I'm creating the support for a new SODAQ board (Sara SFF).
Second, I haven't run the automated tests before, so I don't know what to expect.
The complete run gives me the result below. No need to comment on it. I just wanted to let you know what I have to chew on. I will probably get back with more questions.
ERROR:sodaq-sara-sff:Tests failed: 110 Failures during compilation: - [tests/bench_xtimer](tests/bench_xtimer/compilation.failed) - [tests/pkg_nanopb](tests/pkg_nanopb/compilation.failed) - [tests/suit_manifest](tests/suit_manifest/compilation.failed)
Failures during test: - [examples/micropython](examples/micropython/test.failed) - [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed) - [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed) - [tests/bloom_bytes](tests/bloom_bytes/test.failed) - [tests/cond_order](tests/cond_order/test.failed)
Hi Kees,
Indeed that's a lot of failures!
Regarding the failures during flash, the flashing over USB workflow is rather fragile: if something went wrong (a crash) when testing an application, the board might no recover and cannot be flashed automatically anymore after. There's also the issue with connecting the terminal application to stdio that can timeout sometimes.
The main issues are with the "failures during test" list. I tried some of them on arduino-mkr1000 and also got failures. So there are bugs.
Note that you can run each of them individually using "make BOARD=<your board> -C tests/<test application> flash test". This is what is called by the "compile_and_test_board.py" script.
Regarding "tests/bench_xtimer", you can fix the build issue by adding your board to the LOW_MEMORY_BOARDS list [1]. For the other build failures, I guess some Python packages are missing. Maybe add BUILD_IN_DOCKER=1 to the test command.
Alex
[1] https://github.com/RIOT-OS/RIOT/blob/master/tests/bench_xtimer/Makefile#L7
Thanks Alex,
The tests that need sudo, I've skipped. For now I don't care about these. I was able to rerun quite a few tests by skipping the ones that caused the the board to hang. I also skip all the tests that need sudo. For now I don't care about these.
I'll look at LOW_MEMORY_BOARDS, but if other SAMD21 don't need it, then my board shouldn't need it either.
I also need to look at my execution environment. I don't docker (yet), so maybe I'm using incompatible gcc and/or newlib. I've seen strange problems with puts/printf, where part of the output is missing (and thus I get test timeouts). -- Kees
Hi Kees,
Hi,
I'll look at LOW_MEMORY_BOARDS, but if other SAMD21 don't need it, then my board shouldn't need it either.
All other SAMD21 boards with stdio over USB are in this list so I think yours should be there as well 
I've seen strange problems with puts/printf, where part of the output is missing (and thus I get test timeouts).
Same here, I think there's an issue stdio over USB. If the string to be printed starts with a space or a tab, it's not printed over USB.
Alex
I've create an issue for it, #14184