Hey,
With all the chatter about the CI situation lately I got to wondering if
there might be some way I could help out. It sound like Kaspar has
things under control with the new CI system but what about builders; are
there enough CPU cycles available for the current rate of RIOT
development, what about future development?
For now, it's a *huge* improvement over travis. As dipswitch already
wrote, build times depend a lot on cache hits.
A change somewhere outside of core can take less than 6 minutes to
build, a change to a core header like thread.h basically forces a full
cold-cache rebuild, taking >30 minutes.
Also, a build that is mostly cache-miss fills the ccache (currently set
at 150gb), and ccache has a weird pruning strategy, as the cache fill
rate goes close to 150g, then drops to <50gb, probably pruning more than
necessary.
Not using ccache at all takes ~22 minutes, so I'm considering either not
using ccache, or disabling it when certain conditions apply, like a diff
in core/include.
I think there's quite a bit room for improvement apart from that:
- the cached builds are SSD IO bound, averaging > 5k transactions per
sec, reading >100MB/s while writing ~20MB/s. Putting the whole cache in
memory seems expensive (anyone has spare 256gb in DDR3 DIMMS?), but a
RAID0 of SSD's would probably speed up things.
- the current build system is horribly inefficient.
While that doesn't matter for builds of one app for one MCU, when
building all ~4000 combinations, it matters.
Like, all RIOT's source files get stat'ed 4k times, every compiler check
gets called 4k times, every "(shell ls blah)", .......
I'm working on an improved build system, but it'll be a while until it
is production ready, if we even decide to step away from make
- less drastic would be a rewrite of compile_test.py + "buildtest" make
target, as that is not parallizing very well.
This is also needed for properly distributing the builds, as with the
current scripts, the smallest "chunk" is basically one MCU group, which
on a quad core box probably takes as much time to build as the worst
case 22minutes.
Would it be possible for me and/or other members of the community to
donate CPU time for use as builders?
Sure, as soon as we can make use of it!
If CI builders could be deployed as VMs, or better yet, Docker images
(lighter weight) would it be possible for people like me to help
out?
I agree, Docker is the way to go here.
Kaspar
The latest was that small package changes take about 5 min to complete. If there are changes in the kernel there is a CCACHE huge mismatch and it takes about 30 minutes.