Updates to the build system - modules definition

Development archive
1

Hi Dan, Gaetan,

I wonder, what problems are we trying to solve?

Maybe we can clearly enumerate them first so that we can check later, whether an improved solution really solves these problems.

Cheers,   Thomas

2

Hi,

I think the problem to solve is that RIOT is very difficult to configure.

While working with other IoT operating systems, one can notice that RIOT is very hardly configurable, unless we are very familiar with the build system and specific RIOT CFLAGS and special key-words (USEMODULE, USEPKG, etc…).

Besides, I have no knowledge of a tool that can parse RIOT makefiles and give an insight about module usage, dependencies, features, etc.

Thus IMHO a way to make, first, RIOT more friendly configurable it’s a good and necessary feature. This would also allow to eventually build a dependency tree in a more human-readable fashion, as well as give some help to get different product lines according to the selected features.

However, I’d say that deep modifications to the current build system can lead to instability or buggy builds. I’m not against any improvement to the build system, but maybe as a first approach some additions like files in any more descriptive (parseable) language can help to describe RIOT in a higher level, without affecting at all the underlying system. Then, with the knowledge obtained thanks to such a description, we can imagine to generate makefiles or make “make” calls with the specific RIOT CFLAGS, USEMODULE, etc.

My 2c.

Cheers,

3

Hi Thomas, Francisco, all,

I will try to summarize with three aspects formatted as problems.

Just for clarity, in the following text, use the words ‘description’, ‘informations’, ‘configuration’, ‘dependencies’ to describe all the meta-information that can be attached to a module in addition to its source code.

I also am in demand of other issues that you have and that should be taken into account if the current system is refactored. So I invite you to talk about your case too, I want to start discussions before working on it.

Also I welcome all remarks regarding my problems and the way I want to address them.

1. Configuration is not documented

As Francisco is saying, about configuring an application and so its modules. The problem, for me, is that its hard to know the possible configurations of a module and the impact of the configuration/dependencies.

Details:

When building an application, I need to:

  • describe what I want to use in my application, which usually means saying which module/package I want
  • configure the application and the modules, by using either specific variables DEVELHELP, OUTPUT or manually setting CFLAGS
  • Specify requirements to work only on boards with this hardware support.
  • But it also means understanding the impact of adding a specific module / setting one configuration

Right now the impact is tedious to find. A dependency pulled by one module can have an impact on another module code, on its API, on all modules. And modules configuration, is often only described into the doxygen or the source code: what CFLAGS should I put to configure the radio channel ?

I would like to clearly write down these public configurations and dependencies relations.

2. Information is not readable

The module information cannot be presented and re-interpreted to show on a graphical interface. It is not machine readable, and the same way of defining things has multiple meaning.

Details:

I would need to have parseable files usable by an automated system, which is currently not the case.

For my needs, it would be necessary to refine the current information to make it more precise. The ‘dependencies’ and configurations now used in the build system can be ambiguous to present because they have different meaning.

  • Some optional dependencies have different impact, they:

  • can change the source code behavior

  • can change the module API

  • can affect every modules- The same goes for configuration, is the configuration affecting the module API ?

4

Hi,

5

Pointer?

Cheers   matthias

6

Hi Matthias,

7

Hiho Gaetan,

wasn't it in particular that the current build system does not produce well-defined results in the sense that the outcome depends on the order and hidden interferences??

Wasn't it also that the current configuration system includes overloaded semantics and is thus rather intransparent??

These to me seemed to be the core issues??

Cheers,   Thomas

8

Hi all,

I want to give my opinion on the discussions that happened here in Berlin regarding the build system evolution. And also, choose the direction for next development steps.

# Laze presentation #

Kaspar presented his new tool that replaces the current Make based build system based on a custom python software reading the configuration and generating `ninja` files to builds. It uses YAML description for modules, application, and toolchain elements.

GOOD

* The YAML description is a good direction into what I need as a "descriptive language" for the modules. * Using a meta build-system, removes the need to parse all the configuration from scratch each times an application is build which is a static time consumption for CI. It could be done in make but may be more cumbersome to write and maintain. * Refined module build configuration to only show the needed flags and dependencies, it will help a lot for caching. * I love python.

BAD

* Python is not a build system language, so not designed to do 'targets' generation but more executing commands after the other. * Introducing our own custom build language framework instead of using an existing one, so a non standard tool. * The build system implementation (how to parse yaml files, generate build rules), is not separated from the laze tool itself, so hard to have the implementation stored in the repository while having a generic tool installed globally. * Many argument are based on speed, but the speed improvement are not all a result of using 'laze' only, but cleaning up concepts in general.

# Criticism of the current make build system #

However, I agree that our current build system has a lot of flaws that I would like to see solved.

* Its necessary to run `make clean` all the time because there are changes that do not correctly trigger rebuild, so you are never sure * The module definition/configuration is not structured and descriptive so is problematic to provide finer granularity module building, all modules see all CFLAGS, INCLUDES, and included modules MODULE_ macros. * Its now impossible to cache results between applications as they are done with different config. * Not all dependencies are explicitly described, so hard to understand the optional dependencies, the dependencies on interface, dependencies depending on configuration. This would also help refining module build configuration. * Our current build system only takes care of building one application for one board. For the CI and maintainers, when building several targets, they must re-execute make from scratch for each configuration. * And the initial tool checking, is it linux, modules discovery, should be done every time. And this really adds up on many targets builds * It can only be solved correctly by first having structured module description as you can parse it once and apply it to different configurations * There is no support for building in a common build directory so it takes a lot of space and also rebuilds multiple time the same thing. * It currently makes no sense, as module have different configurations, but after fixing the previous issues, one built module configuration could be shared between different apps, saving space and compile time.

# Next steps together #

To go forward and use what he developed as a possible solution, I proposed to start a documentation phase, on both his new system + YAML module format, and on what and how things are currently done in the build system. Adding user and developer documentation for the current system would fill the lack of documentation and also help describing all the currently existing features, show the problems, find forgotten requirements. And documenting his description format and his tool could show what are the benefits of his choices and also the tradeoffs.

One of my question is, is it interesting to integrate some changes in the current make system, like some early module description, even if replaced later by another build system ?

I will open issues on github to follow the next steps.

Cheers, Gaëtan