Good point. I think many build systems have the implicit assumption that (re)configuration of the build e.g. for a new board or module set is rarely done. That assumption will work well on the developers machines, but not at all for the CI.
Caching configurations might be a way out, but would come with significant complexity for correctly invalidating cache entries on dependency / configuration changes. And caches have the disadvantage that they can be cold when a system is restartet, e.g. to boot a new kernel.