Using git capabilities in the modular project build system
In our blog, we already talked about the principles of organizing the repository of a large project as a set of independent modules, which allows you to organize the extraction of source codes into an arbitrary file structure of the working copy. Of course, such an approach could not but affect the project build system, since it demanded the creation of a mechanism for tracking dependencies between modules, taking into account their actual placement. This article focuses on how git can be used to solve not only this problem, but also to extract a project fragment with automatic consideration of internal intermodule dependencies.
It was originally intended to provide this publication with fragments of the build system as implemented in Linter , however, we do not use native git modules in this project and use make our own development utility. Therefore, in order not to hurt the practical value of the material for readers, all examples are adapted for use in the bundle git submodules and gnu make, which led to certain difficulties, which will be indicated below.
In order to simplify, we will consider the integration of the build system with git on the example of a conditional product called project, which consists of the following functional modules:
applications - the application itself;
demo - demo examples;
libfoo and libbar are the libraries on which applications depend.
The project dependency graph is as follows:
Figure 1: Project dependency graph
')
From the point of view of the version storage system, the project is divided into five separate repositories — four for modules and a fifth, project.git, which serves as a container and contains an assembly system. This method of organization has several advantages in comparison with the mono-repository:
Despite the fact that the recursive approach to the organization of the assembly system is justifiably criticized , it still allows you to significantly reduce the cost of supporting the project, so in our example we will use it. At the same time, the root makefile of the project should not only “know” the position of the modules within the project, but also ensure that the child make-processes are invoked in order in the right order: from the branches of the dependency tree to the roots. To do this, you should explicitly describe these inter-module dependencies; in our example, this is done as follows :
A correct traversal of this tree can be provided by means of make, creating dynamic targets with explicit dependencies, for which we will declare a function gen-dep of the following form:
Now, if in the body of the root Makefile call gen-dep for all modules
then it will generate the following dynamic targets at runtime (this can be checked by running make with the -p key)
that allows when referring to them to ensure the call dependencies in the right order. At the same time, if the name of the target matches the existing file or directory, then this can disrupt execution, since make “does not know” that these our goals are actions, and not files, in order to avoid this we will explicitly indicate:
Suppose that the developer has the task of making changes to the application, for which he needs to get only the submodules application, libbar, libfoo. For this, the build system should, based on the dependencies declared above, form a description of the modules and their placement for later use by git, which, as you know , describes the registered submodules in the file named .gitmodules located in the root of the cloned repository.
We make the following changes to our example to ensure the generation of .gitmodules of the minimum required composition:
Now our conditional developer, having called make application, can create the following file:
which can already be modified and parsed by git-a , for example in the following way:
By itself, the presence of the .gitmodules file in the root of the repository does not register the modules in the index, so until the moment of initialization and cloning of the submodules, all necessary corrections can be made to the file.
As for the initialization of submodules directly, the first serious inconvenience in the implementation of native modules in git is manifested here: this version control system stores metadata about modules both in the index and in the .gitmodules file. Looking into the source code it becomes clear that we have two not the best alternatives.
The first is to add the information about the modules to the index as follows :
in this case, it is possible to work with submodules using the standard git-submodule (iterators, group operations, etc.), however, moving / deleting modules, as well as their branching, will require additional auxiliary operations. This situation was one of the reasons why we refused to use git-submodules in the Linter repository. An alternative to submodule add is cloning modules without registering with the index, which can be done as follows:
in this case, it is necessary to explicitly specify all $ path in .gitignore, otherwise git will take the cloned submodules as regular directories and process them and contents as untraceable files.
Anyway, after cloning by any of the above methods, the working copy will correspond to the situation of extracting the selected tree fragment.
Figure 2: Project dependency graph. A selectable module tree is highlighted by a fill.
and, provided that the intermodular dependencies are correctly declared, it contains everything necessary for compiling the application.
Another task that the assembly system solves is to determine the current position of the modules. For this, we will use the file descriptor generated by us earlier. As in the case of initialization, there are several options. The simplest thing is to take advantage of git config features:
This solution is not ideal from the point of view of portability, but another option is available only if you use GNU make version 4 or higher — in this case, parsing the .gitmodules file can be implemented using the GNU make extensions.
Let us remind ourselves once again that the example available on github is an adaptation of our linmodules + linflow bundle-based solutions for gitmodules + GNU make, so some of the drawbacks of pairing these tools are not solved in the most elegant way, and the calls of child make files in modules are replaced with “dummy” ".
Nevertheless, the mechanism proved itself quite well when working with a large project and successfully “copes” with the repository of 102 git submodules, between which there are 308 intermodule connections (both logical and assembly) with a bond graph diameter of 5 units (see illustration above).
Some words about examples
It was originally intended to provide this publication with fragments of the build system as implemented in Linter , however, we do not use native git modules in this project and use make our own development utility. Therefore, in order not to hurt the practical value of the material for readers, all examples are adapted for use in the bundle git submodules and gnu make, which led to certain difficulties, which will be indicated below.
Demo Description
In order to simplify, we will consider the integration of the build system with git on the example of a conditional product called project, which consists of the following functional modules:
applications - the application itself;
demo - demo examples;
libfoo and libbar are the libraries on which applications depend.
The project dependency graph is as follows:
Figure 1: Project dependency graph
')
Storage organization
From the point of view of the version storage system, the project is divided into five separate repositories — four for modules and a fifth, project.git, which serves as a container and contains an assembly system. This method of organization has several advantages in comparison with the mono-repository:
- each submodule has a separate revision history;
- the ability to clone only part of the project;
- each repository can have individual rules and access policies;
- the ability to extract the project into an arbitrary structure of the working copy.
Submodules and dependencies
Despite the fact that the recursive approach to the organization of the assembly system is justifiably criticized , it still allows you to significantly reduce the cost of supporting the project, so in our example we will use it. At the same time, the root makefile of the project should not only “know” the position of the modules within the project, but also ensure that the child make-processes are invoked in order in the right order: from the branches of the dependency tree to the roots. To do this, you should explicitly describe these inter-module dependencies; in our example, this is done as follows :
MODS = project application libfoo libbar demo submodule.project.deps = application demo submodule.demo.deps = application submodule.application.deps = libfoo libbar submodule.libfoo.deps = submodule.libbar.deps = A correct traversal of this tree can be provided by means of make, creating dynamic targets with explicit dependencies, for which we will declare a function gen-dep of the following form:
define gen-dep $(1):$(foreach dep,$(submodule.$(1).deps),$(dep)) ; endef Now, if in the body of the root Makefile call gen-dep for all modules
$(foreach mod,$(MODS),$(eval $(call gen-dep,$(mod)))) then it will generate the following dynamic targets at runtime (this can be checked by running make with the -p key)
project: application demo demo: application application: libfoo libbar libbar: libfoo: that allows when referring to them to ensure the call dependencies in the right order. At the same time, if the name of the target matches the existing file or directory, then this can disrupt execution, since make “does not know” that these our goals are actions, and not files, in order to avoid this we will explicitly indicate:
$(eval .PHONY: $(foreach mod,$(MODS), $(mod))) Suppose that the developer has the task of making changes to the application, for which he needs to get only the submodules application, libbar, libfoo. For this, the build system should, based on the dependencies declared above, form a description of the modules and their placement for later use by git, which, as you know , describes the registered submodules in the file named .gitmodules located in the root of the cloned repository.
We make the following changes to our example to ensure the generation of .gitmodules of the minimum required composition:
… MODURLPREFIX ?= git@git-common.relex.ru/ MODFILE ?= .gitmodules … define tmpl.module "[submodule \"$(1)\"]" endef define tmpl.path "\tpath = $(1)" endef define tmpl.url "\turl = $(1)" endef … define submodule-set submodule.$(1).name := $(2) submodule.$(1).path := $(3) submodule.$(1).url := $(4) endef define set-default $(call submodule-set,$(1),$(1),$(1),$(MODURLPREFIX)$(1).git) endef define gen-dep $(1):$(foreach dep,$(submodule.$(1).deps),$(dep)) @echo "Register module $(1)" @echo $(call tmpl.module,$(submodule.$(1).name)) >> $(MODFILE) @echo $(call tmpl.path,$(submodule.$(1).path)) >> $(MODFILE) @echo $(call tmpl.url,$(submodule.$(1).url)) >> $(MODFILE) endef … $(foreach mod,$(MODS),$(eval $(call set-default,$(mod)))) Now our conditional developer, having called make application, can create the following file:
[submodule "libfoo"] path = libfoo url = git@git-common.relex.ru/libfoo.git [submodule "libbar"] path = libbar url = git@git-common.relex.ru/libbar.git [submodule "application"] path = application url = git@git-common.relex.ru/application.git which can already be modified and parsed by git-a , for example in the following way:
git config -f .gitmodules --get submodule.application.path application By itself, the presence of the .gitmodules file in the root of the repository does not register the modules in the index, so until the moment of initialization and cloning of the submodules, all necessary corrections can be made to the file.
As for the initialization of submodules directly, the first serious inconvenience in the implementation of native modules in git is manifested here: this version control system stores metadata about modules both in the index and in the .gitmodules file. Looking into the source code it becomes clear that we have two not the best alternatives.
The first is to add the information about the modules to the index as follows :
#!/bin/sh git config -f .gitmodules --get-regexp '^submodule\..*\.path$' | while read path_key path do url_key=$(echo $path_key | sed 's/\.path/.url/') url=$(git config -f .gitmodules --get "$url_key") git submodule add --force $url $path done in this case, it is possible to work with submodules using the standard git-submodule (iterators, group operations, etc.), however, moving / deleting modules, as well as their branching, will require additional auxiliary operations. This situation was one of the reasons why we refused to use git-submodules in the Linter repository. An alternative to submodule add is cloning modules without registering with the index, which can be done as follows:
#!/bin/sh git config -f .gitmodules --get-regexp '^submodule\..*\.path$' | while read path_key path do url_key=$(echo $path_key | sed 's/\.path/.url/') url=$(git config -f .gitmodules --get "$url_key") git clone $url $path done in this case, it is necessary to explicitly specify all $ path in .gitignore, otherwise git will take the cloned submodules as regular directories and process them and contents as untraceable files.
Anyway, after cloning by any of the above methods, the working copy will correspond to the situation of extracting the selected tree fragment.
Figure 2: Project dependency graph. A selectable module tree is highlighted by a fill.
and, provided that the intermodular dependencies are correctly declared, it contains everything necessary for compiling the application.
Positioning modules
Another task that the assembly system solves is to determine the current position of the modules. For this, we will use the file descriptor generated by us earlier. As in the case of initialization, there are several options. The simplest thing is to take advantage of git config features:
define get-path $(shell git config -f .gitmodules --get "submodule.$(1).path") endef define get-url $(shell git config -f .gitmodules --get "submodule.$(1).url") endef This solution is not ideal from the point of view of portability, but another option is available only if you use GNU make version 4 or higher — in this case, parsing the .gitmodules file can be implemented using the GNU make extensions.
Conclusion
Let us remind ourselves once again that the example available on github is an adaptation of our linmodules + linflow bundle-based solutions for gitmodules + GNU make, so some of the drawbacks of pairing these tools are not solved in the most elegant way, and the calls of child make files in modules are replaced with “dummy” ".
Nevertheless, the mechanism proved itself quite well when working with a large project and successfully “copes” with the repository of 102 git submodules, between which there are 308 intermodule connections (both logical and assembly) with a bond graph diameter of 5 units (see illustration above).
Source: https://habr.com/ru/post/260311/
All Articles