Build system for large modular projects
On the pages of our blog, we have already written about the advantages of organizing a repository of a large project in a way that suggests the possibility of extracting sources into the variable structure of the working copy. The use of this approach, coupled with the needs of simple configuration, fragmentary assembly, support of several dozen operating systems for a wide range of hardware platforms, have led us to develop our own assembly system. This article talks about the solutions we found that might be of interest to developers facing difficulties in supporting the infrastructure of large projects.
Before going directly to the technical details should be noted two important points. First, the system works on top of the linmake make-utility we developed, the features of which will be discussed separately. And, secondly, the development was carried out to solve the problems of the production of DBMS Linter ( www.linter.ru ), which brought some specificity, but not so essential that the solution could not be adapted to any project.
As is often the case, the development and complexity of the project at some point led to the fact that supporting the infrastructure of the assembly became too expensive and this was facilitated by several reasons, the complete enumeration of which would take indecently much space, so let's allow ourselves to select only those that caused more Number of complaints from project participants:
Of course, in addition to the problems, there were also wishes for the implementation of new "features", so when the decision was made to develop a new unified assembly system, which was called unimake, we quite clearly had no idea what goals we wanted to achieve:
The build is made in a different directory ( srcroot ) directory - build directory ( bldroot ). Each project build is entirely determined by a set of sets:
Project configuration option
The combination of the listed parameters determines all possible options that are pre-filtered by the assembly system in order to weed out unnecessary and meaningless combinations.
In turn, each module expands the parameters βfor itselfβ with the help of two file descriptors: for the module and for the assembly process, which are written in a declarative style and do not contain rules (with rare exception). The module descriptor contains general information about the module: name and version, supported platforms, compilers and architectures, thread models, targets. All declarations (except the name) are optional and if they are missing, the default values ββare used.
')
Option module descriptor
The assembly descriptor declares the goals, their composition, directives, search directories, external and internal dependencies of the module.
Build Descriptor Option
The bldroot directory structure repeats the srcroot to the root level of each module ( modsrc ), but already in them contains all the actual options defined by valid combinations of general design and modular configurations. Under each of these options, a directory of the form $ (MODULE) / $ (PLT) _ $ (ARCH) _ $ (CMPL) $ (CMPLV) _ $ (TYPE) _ $ (CFG) is created (for example, example / LINUX_AMD64_GCC4_MD_R_base60), we will call further these directories as modbld .
Variant content modsrc
Modbld content variation
In each valid modbld , three files are created during the directory traversal: compiler options (* .cfl in our case), linker options (* .lnk - in the example) and auxiliary makefile, which are designed to compile and link targets that bypass the common system assembly, which is often required for debugging tasks. Thus, there are two options for using the system:
The call flow for both cases is shown in the illustrations below.
Figure 1: Building the whole project (1) results in the formation of a sequence of calls to the root makefile (3) for all possible combinations of the build options (2). As a result of filtering (3), obviously unsuitable options are eliminated. The files are module descriptors, (4) based on dependencies and additional parameters adjust the options. The assembly descriptors (5) fulfill the rules (6) and form the target directories with the results of execution (7).
Figure 2: Updating existing modules (1) works in a simplified way: the auxiliary rules in modbld (3) update (4) their goals without using a module handle and filters.
As mentioned above, all the rules are placed in a separate module ( unimake ) at the project level, which, in addition to the assignments of the rules themselves, is responsible for storing the dependency tree between modules. At the same time, each module of the declared generates a separate target with the generated dependent targets.
Storage and use of dependencies between modules
Thanks to the built-in parser of files for the placement of modules, linmodules have the ability to track the current position of the modules in the source tree and use a simple path definition.
Reading and registering modules and paths
The approach described in the previous section was implemented by us for the infrastructure of the Linter project. And, despite the fact that it happened relatively recently (about six months ago), the system has already positively established itself in terms of ease of use, scalability and performance.
Even in the early stages of implementation, we encountered the well-known flaws of gnu make, so the solution is based on a proprietary make-utility - linmake, the syntax of which shows all the listings in this article. Most likely, in the foreseeable future we will return to the topic of linmake and its features on the blog pages, but so far this has not happened the publication of the system as it is used in the development does not make sense. However, it would be wrong to deprive the reader of the opportunity to test the proposed model, so a working prototype for gnu make is available here ( github.com ).
Before going directly to the technical details should be noted two important points. First, the system works on top of the linmake make-utility we developed, the features of which will be discussed separately. And, secondly, the development was carried out to solve the problems of the production of DBMS Linter ( www.linter.ru ), which brought some specificity, but not so essential that the solution could not be adapted to any project.
Why was it necessary to create a new build system?
As is often the case, the development and complexity of the project at some point led to the fact that supporting the infrastructure of the assembly became too expensive and this was facilitated by several reasons, the complete enumeration of which would take indecently much space, so let's allow ourselves to select only those that caused more Number of complaints from project participants:
- due to the fact that back in 1999 there was no acceptable cross-platform tool, we had to maintain two parallel build systems for a long time: based on wmake for windows and make for * nix;
- the variety of supported UNIX-like platforms has led to an increase (and therefore complication) of compilation and linking options in project modules;
- in turn, the windows version build was complicated by the need to support a large number of compilers;
- there was no simple description and resolution mechanism as external and internal project dependencies.
Of course, in addition to the problems, there were also wishes for the implementation of new "features", so when the decision was made to develop a new unified assembly system, which was called unimake, we quite clearly had no idea what goals we wanted to achieve:
- the system should work uniformly on all supported platforms;
- a change in the position of the module (hereinafter, under the module we will understand the functionally self-sufficient part of the project) in the working tree should not affect the performance;
- a simple mechanism is needed to add new target platforms, architectures, compilers and their versions;
- should be stored as standard configurations for versions and revisions of the product, and provide the ability to customize them if necessary;
- need a simple way to automatically account for external and internal dependencies in the project, which would automatically determine the order of operations;
- the system should provide the ability to easily build a part of the project with all its dependencies.
Assembly model, general provisions
The build is made in a different directory ( srcroot ) directory - build directory ( bldroot ). Each project build is entirely determined by a set of sets:
- configurations / product versions (CONFIGS);
- target platforms (PLATFORMS);
- target architectures (ARCHS);
- compilers (COMPILERS);
- compiler versions ($ (CMPL) _VERS);
- build platform (HOST.PLT);
- build platform architecture $ (HOST.ARCH).
Project configuration option
... CONFIGS = base60 full60 PLATFORMS = LINUX ARCHS = AMD64 JAVA .NET COMPILERS = GCC JAVAC MONO JAVAC_VERS = 1.4 1.5 1.6 GCC_VERS = 4 MONO_VERS = 3 β¦ HOST.PLT = LINUX HOST.ARCH = AMD64 DEBUG = RELEASE The combination of the listed parameters determines all possible options that are pre-filtered by the assembly system in order to weed out unnecessary and meaningless combinations.
In turn, each module expands the parameters βfor itselfβ with the help of two file descriptors: for the module and for the assembly process, which are written in a declarative style and do not contain rules (with rare exception). The module descriptor contains general information about the module: name and version, supported platforms, compilers and architectures, thread models, targets. All declarations (except the name) are optional and if they are missing, the default values ββare used.
')
Option module descriptor
MODULE = example # VERSIONS = # VERSIONS_REQ:= $(CFG.VER) # LINK_TYPES = static dynamic # / THREAD_TYPES = mt # DST_SRC = example.h # DONT_BUILD_WATCOM = # , β watcom ( ) DONT_BUILD_WINCE = # β WinCE The assembly descriptor declares the goals, their composition, directives, search directories, external and internal dependencies of the module.
Build Descriptor Option
... TARGET = $(MODULE) # , + , (.so, .a, .dll ..) DEFINES = _VER=$(CFG_VER) SOME_DEFINES # DEFINES_WINNT = EXAMPLE_WIN # Windows DEFINES_UNIX = EXAMPLE_POSIX # *nix CDIR = $(MODROOT);$(MODROOT)/utils; # INCLDIR = $(MODROOT);$(ANOTHER_MOD); # OBJS = & example.obj # OBJS_UNIX = & charset.obj # *nix SLIBS_WINNT = $(ANOTHER_LIB) oldnames # windows ... SLIBS_UNIX = $(ANOTHER_LIB) # *nix ... The bldroot directory structure repeats the srcroot to the root level of each module ( modsrc ), but already in them contains all the actual options defined by valid combinations of general design and modular configurations. Under each of these options, a directory of the form $ (MODULE) / $ (PLT) _ $ (ARCH) _ $ (CMPL) $ (CMPLV) _ $ (TYPE) _ $ (CFG) is created (for example, example / LINUX_AMD64_GCC4_MD_R_base60), we will call further these directories as modbld .
Variant content modsrc
<srcroot> βββ example βββ example.c βββ example.h βββ makefile.lmk βββ makelibs Modbld content variation
<bldroot> βββ example βββ LINUX_AMD64_GCC4_MD_R_base60 β βββ charset.obj β βββ example.cfl β βββ example.h β βββ example.lnk β βββ example.obj β βββ example.so β βββ makefile βββ LINUX_AMD64_GCC4_MD_R_full60 β βββ charset.obj β βββ example.cfl β βββ example.h β βββ example.lnk β βββ example.obj β βββ example.so β βββ makefile βββ LINUX_AMD64_GCC4_MT_R_base60 β βββ charset.obj β βββ example.a β βββ example.cfl β βββ example.h β βββ example.lnk β βββ example.obj β βββ makefile βββ LINUX_AMD64_GCC4_MT_R_full60 βββ charset.obj βββ example.a βββ example.cfl βββ example.h βββ example.lnk βββ example.obj βββ makefile In each valid modbld , three files are created during the directory traversal: compiler options (* .cfl in our case), linker options (* .lnk - in the example) and auxiliary makefile, which are designed to compile and link targets that bypass the common system assembly, which is often required for debugging tasks. Thus, there are two options for using the system:
- assembly of the entire project / module for the first time;
- update module.
The call flow for both cases is shown in the illustrations below.
Figure 1: Building the whole project (1) results in the formation of a sequence of calls to the root makefile (3) for all possible combinations of the build options (2). As a result of filtering (3), obviously unsuitable options are eliminated. The files are module descriptors, (4) based on dependencies and additional parameters adjust the options. The assembly descriptors (5) fulfill the rules (6) and form the target directories with the results of execution (7).
Figure 2: Updating existing modules (1) works in a simplified way: the auxiliary rules in modbld (3) update (4) their goals without using a module handle and filters.
As mentioned above, all the rules are placed in a separate module ( unimake ) at the project level, which, in addition to the assignments of the rules themselves, is responsible for storing the dependency tree between modules. At the same time, each module of the declared generates a separate target with the generated dependent targets.
Storage and use of dependencies between modules
β¦ dep_example = another dep_another = β¦ module-deps = $(foreach name,$(DEP_$(1)), $(MOD_$(name))) gen-module-deps = $(foreach name,$(DEP_$(1)), $(2)_$(MOD_$(name))) !define gen-target $(1): .SYMBOLIC @$(MAKE) MODULE=$(1) !endef !define gen-targets TARGETS_$(1) := $(foreach mod,$(ALL_MODULE_NAMES), $(1)_$(mod)) $(1): $$(TARGETS_$(1)) @%null !endef gen-targets-without-deps = $(foreach mod,$(ALL_MODULE_NAMES),$(gen-target ,$(mod))) !eval $(gen-targets-without-deps) !eval $(gen-targets dep) Thanks to the built-in parser of files for the placement of modules, linmodules have the ability to track the current position of the modules in the source tree and use a simple path definition.
Reading and registering modules and paths
#git modules LINMODS=$(modlist $(SRCROOT)/.linmodule) !define add-mod MOD_$(1) = $$(modpath $(1)) !endef !eval $(foreach i,$(LINMODS),$(add-mod $(i))) Implementation
The approach described in the previous section was implemented by us for the infrastructure of the Linter project. And, despite the fact that it happened relatively recently (about six months ago), the system has already positively established itself in terms of ease of use, scalability and performance.
Even in the early stages of implementation, we encountered the well-known flaws of gnu make, so the solution is based on a proprietary make-utility - linmake, the syntax of which shows all the listings in this article. Most likely, in the foreseeable future we will return to the topic of linmake and its features on the blog pages, but so far this has not happened the publication of the system as it is used in the development does not make sense. However, it would be wrong to deprive the reader of the opportunity to test the proposed model, so a working prototype for gnu make is available here ( github.com ).
Source: https://habr.com/ru/post/265001/
All Articles