RIOT CI megathread

See artifacts in e.g. https://github.com/RIOT-OS/RIOT/actions/runs/252967164 for example out put and compile_and_test_for_board.py: add optional JUnit XML support · RIOT-OS/RIOT@4cc6963 · GitHub for the actual code change.

using junit, I had trouble representing actual node output, due to control characters in there. is that solved somehow?

Special characters are escaped by that library.

For some stuff, like ping6 -f in the release specs (not using that library but the in-build support for junit XML in pytest) I also had problems, because the sheer number of output with that, basically made the XML huuuuge. I solved it by piping away the chars in question.

Also nice would be to expand on the benchmark collection feature. E.g. if we notice that performance for some benchmark did drop recently, it would be nice to let the CI run a specific application on a specific commit with a given number of repetitions. (No re-flash between repetitions, just re-run the test.)

PRs can already be flagged as such using the “CI: skip compile tests” label.

yeah, we have to rephrase, @chrysn meant “assert that PR does not change code”

Ah sorry, should have read the whole discussion. I think “assert” is also the wrong word (I was thinking of assert(code does not change) first). “Check that PR does not change code”?

Isn’t that equivalent? :slight_smile:

Not really. An assertion is a statement made beforehand about some pre-condition. A check is a test of a certain condition. What you described is a check, not an assertion (which is more what the “CI: skip compile test” label does, it asserts there is no need to recompile)

8 posts were split to a new topic: CI: code coverage

So to rephrase my suggestion in light of the disussion: I’d like to have a “CI: assert identity after compilation” flag.

(This entails that any later test, especially HIL, can be skipped).

1 Like

I got another one: prevent semantic conflicts.

@aabadie suggested collecting code coverage.

I’ve moved the resulting discussion to a new topic:

Another one for the wishlist, I would like Murdock to comment on PRs with results. The message content should be helpful to both regular contributors and newcomers. Among other things this could contain the warnings from the static tests such as the codespell warnings.

I think this could help by making it easier for the regulars to not forget the static test warnings. For newcomers it pulls their attention to the CI and the message content can explain or hint to what should be fixed.

1 Like

What would also be nice is an “abort on static_test failure” flag. E.g. when I’m sure that the code is working, I don’t want to waste a full murdock cycle on fixing an incorrectly spelled word or a whitespace error.

What would also be nice is an “abort on static_test failure” flag

Or just drop the static tests from Murdock and only run Murdock when the static tests are green and the Ready for CI build label is set. The problem with doing so is with the pr_check check that will fail with fixup commits. So in this case, Murdock could only be triggered once the PR is squashed and we don’t want that. Maybe we could have the pr_check (with commit comments and fixup commit checks) in a different action.

Or we combine it with what currently is the check-label action. One job to check the labels, one to check the commit messages.

I’m very much for having one authorative CI, compared to checking labels with github-actions, checking docs with circle ci and checking compilation/tests with Murdock…

I disagree. We should follow the Unix philosophy here (“Make each program do one thing well”). The different CI solutions that we use come with different strength:

  • Murdock is very good for building,
  • Circle CI can build the documentation and provide a preview link,
  • Github actions are well integrated with Github (labels) and very flexible.

A single CI that proposes just the 3 points above doesn’t exist and having one authoritative CI, like Murdock, that fulfills them would need some work each time a new feature needs to be developed/configured and would come with (usability) limitations. For example, Murdock already fulfills points 1 and 3 (without the Docker container flexibility) but not point 2 (documentation preview is not available). Adding this last feature can be quite some work, adding it to Circle CI was a breeze and requires no infrastructure maintenance.

The diversity that we have now (static tests/basic builds/python tests on GH actions, doc build/preview on Circle CI, massive builds on Murdock) goes into the right direction: use the best solutions that exist for a given problem and avoid re-inventing the wheel.

The next step is to remove the static tests from Murdock and to improve the workflow that allows to trigger Murdock, as discussed above.

I think it is better to reduce the different CI tools if feasible. It may not always be though.

The things to consider are the level of expertise required to accomplish the task vs the level of expertise we have. Having fewer CI tools means we may require more expertise but that effort is not distributed across many different tools. The bus factor should also be considered. It is a benefit to have a few people understand one tool rather than everyone owning their own.

That being said something like Circle CI doing the documentation is simple enough that anyone can come in without much effort to maintain or correct it. Maybe it would make sense to move the documentation task to GH actions or something but as of now I think it is not critical.

Something like what Murdock does is niche and non-trivial to implement on other CI tools.

Maybe we should first loosely define a list of tasks.

  • building
    • binaries
    • documentation
  • testing
  • serving
    • build results
    • test results
    • documentation
  • static checks
  • github checks etc…