Git subrepo

The git-subrepo project has been around for a long time, but there is an unfair mention of it. The author of git-subrepo is Ingy döt Net .

If you look at the history of the komits of the master branch of the project, it may seem that the project stopped developing 2 years ago. However, work on the project is underway and I would like to hope that version 0.4.0 will be released soon.

An important feature of this tool is that on the user side there is no need to install git-subrepo until the user decides to do commit in the upstream-repository of subprojects. In addition, the user receives a fully prepared and customized source tree at the time of copying the main repository using the standard git-clone (1) command.

When choosing the means of supporting submodules / subtrees / subprojects of the main container repository, the developer first of all determines the range of possibilities that a particular mechanism provides and answers the following questions:

• whether it is necessary to store the full history of the subproject in the main repository or enough squashed commits;
• whether the ability to deliver changes from the subproject to the upstream repository of the subtree is needed;
• is there a need to connect fixed tags to the upstream repository of the subproject or is it enough to connect the branches;
• whether it will be necessary in the future to delete both the subprojects themselves and, which have become unnecessary, a part of the history of these subprojects;
• whether the user will have to take any actions to manually configure the subprojects after cloning the repository of the main project;
• how time-consuming will be the question of analyzing the history of connecting subprojects and specific audits, from which the subproject originates;
• how a particular tool will affect the configuration management policy (Source Configuration Management) and how much this tool will complicate the daily work of engineers.

Of course, this list of questions cannot reflect the full completeness of the input parameters necessary for the correct choice, but for a preliminary review of existing tools, it is quite sufficient and, speaking of the git-subrepo project , we urge the reader to consider this project from these positions.

Git-subrepo installation

The git-subrepo package , on the developer’s side, can be installed both locally, in its home directory, and at the system level.

In the first case, it is enough to clone the git-subrepo repository into the desired directory, for example, ~ / bin :

bash-4.4$ cd ~/bin bash-4.4$ git clone https://github.com/ingydotnet/git-subrepo.git 

and set up environment variables

 bash-4.4$ vim subrepo-env.sh #!/bin/sh export GIT_SUBREPO_ROOT="/home/username/bin/git-subrepo" export PATH="/home/username/bin/git-subrepo/lib:$PATH" export MANPATH="/home/username/bin/git-subrepo/man:$MANPATH" :wq bash-4.4$ source ./subrepo-env.sh 

If you look at the variables defined in the git-subrepo Makefile :

 # Install variables: PREFIX ?= /usr/local INSTALL_LIB ?= $(DESTDIR)$(shell git --exec-path) INSTALL_EXT ?= $(INSTALL_LIB)/$(NAME).d INSTALL_MAN1 ?= $(DESTDIR)$(PREFIX)/share/man/man1 

it is easy to find out that at the system level, git-subrepo is installed in the directory where Git is located:

 bash-4.4$ bash-4.4$ git --exec-path /usr/libexec/git-core bash-4.4$ 

Thus, the command for installing git-subrepo may look, for example, as follows:

 bash-4.4$ make PREFIX=/usr install 

The presence of the DESTDIR variable allows us to make a package for any Linux distribution without any additional effort (of course, if we are not in a cross-environment), which can be useful for DevOps engineers.

Install git-subrepo on behalf of the superuser:

 bash-4.4$ bash-4.4$ cd git-subrepo/ bash-4.4$ make PREFIX=/usr install install -C -d -m 0755 /usr/libexec/git-core/ install -C -m 0755 lib/git-subrepo /usr/libexec/git-core/ install -C -d -m 0755 /usr/libexec/git-core/git-subrepo.d/ install -C -m 0755 lib/git-subrepo.d/help-functions.bash lib/git-subrepo.d/bash+.bash /usr/libexec/git-core/git-subrepo.d/ install -C -d -m 0755 /usr/share/man/man1/ install -C -m 0644 man/man1/git-subrepo.1 /usr/share/man/man1/ bash-4.4$ 

To analyze the capabilities of git-subrepo, we need a simple test environment, where we can reproduce standard scripts.

Test environment

We will create three directories owner , remote , user , in which we will place the upstream and local repositories of the developer and user.

 bash-4.4$ vim _init.sh #!/bin/sh CWD=`pwd` mkdir remote owner user cd remote git init --bare build-system.git git init --bare platform.git cd ../owner git clone $CWD/remote/build-system.git git clone $CWD/remote/platform.git cd build-system echo -e "\n[master] build-system 1.0.0\n" >README git add README git commit -m "init build-system master 1.0.0" git push cd ../platform echo -e "\n[master] platform 1.0.0\n" >README git add README git commit -m "init platform master 1.0.0" git push cd ../../user git clone $CWD/remote/build-system.git git clone $CWD/remote/platform.git cd $CWD :wq bash-4.4$ bash-4.4$ ./_init.sh bash-4.4$ 

Here,

The author of the project and users have their own copies of upstream-repositories on their machines, presented in our example in the owner and user directories, respectively.

The author’s task is to include the builld-system subproject in the main platform tree to provide the following possibilities to users and project participants:

• clone the main repository with the build-system subproject included in it and at the same time do not worry about setting up versions or revisions. That is, each branch of the platform repository must correspond to a specific revision of a specific build-system repository branch and the user must receive the configured source tree in one git-clone (1) operation, without any additional actions.
• deliver your changes to the upstream project repositories, both to the main and auxiliary ones.
• receive changes made by other project participants or users, of course, if they have the appropriate rights.

Consider the actions of the author, which he must carry out to implement these requirements.

Subproject connection

To connect a new subtree, use the git subrepo clone command, which is similar in its purpose to the git-clone (1) command. The required parameter of the command is the URL of the remote repository. You can also specify the directory in which the subproject will be located and the branch of the remote repository. We will work with master branches, therefore, in our example, we omit unnecessary command parameters.

So, the author of the project can connect the build-system subproject using his git subrepo clone ../../remote/build-system.git/ build-system command :

 bash-4.4$ bash-4.4$ cd owner/platform/ bash-4.4$ git subrepo clone ../../remote/build-system.git/ build-system Subrepo '../../remote/build-system.git' (master) cloned into 'build-system'. bash-4.4$ 

Consider the changes that occurred in the local platform repository:

 bash-4.4$ bash-4.4$ git status On branch master Your branch is ahead of 'origin/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working tree clean bash-4.4$ bash-4.4$ bash-4.4$ git subrepo status 1 subrepo: Git subrepo 'build-system': Remote URL: ../../remote/build-system.git Upstream Ref: b2f5079 Tracking Branch: master Pulled Commit: b2f5079 Pull Parent: b5e76a7 bash-4.4$ 

The history of the build-system subproject is not supplied to the main tree, we have only one squashed committee, which is accompanied by reference information. This information comes under version control in the ./build-system/.gitrepo/config file:

 [subrepo] remote = ../../remote/build-system.git branch = master commit = b2f507918f2821cb7dd90c33223ed5cc3c9922a2 parent = b5e76a713f895565b06ee3ccfa29f19131ba06dd method = merge cmdver = 0.4.1 

Information about subprojects can be obtained using the git subrepo config command , for example, you can learn the revision of the remote / build-system.git upstream project that just came to the main repository using the command:

 bash-4.4$ bash-4.4$ git subrepo config build-system commit Subrepo 'build-system' option 'commit' has value 'b2f507918f2821cb7dd90c33223ed5cc3c9922a2'. bash-4.4$ 

It should be mentioned that the original git-subrepo package stores information about the subprojects not in the .gitrepo / config file, but in the .gitrepo file. If you prefer to store information in a hidden subdirectory, you can select the git-subrepo-0.3.2 branch or the git-subrepo-0.4.1 branch of the git-subrepo child repository .

So, we got the latest version of the remote / build-system.git upstream-repository master branch and placed it in the build-system subdirectory of the main project platform .

To deliver these changes to the remote / platform.git upstream repository, the author needs to run the git push command:

 bash-4.4$ bash-4.4$ git push Enumerating objects: 7, done. Counting objects: 100% (7/7), done. Delta compression using up to 4 threads Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 849 bytes | 849.00 KiB/s, done. Total 6 (delta 0), reused 0 (delta 0) To /home/prog/0.4.1/remote/platform.git <font color="#8b0000">b5e76a7..6b831e4</font> master -> master bash-4.4$ 

More information about the git subrepo commands can be obtained from the ReadMe.pod file or on the command line.

 bash-4.4$ git subrepo help <command> 

eg:

 bash-4.4$ git subrepo help clone 

Now let's consider everything that happens on the part of the user.

Getting the code by users

At the moment, when the user has not yet received updates to the upstream repository platform.git , his copy contains one README file

 bash-4.4$ bash-4.4$ cd user/platform/ bash-4.4$ ls README bash-4.4$ 

containing one line:

 bash-4.4$ bash-4.4$ cat README [master] platform 1.0.0 bash-4.4$ 

After removing changes to the upstream repository

 bash-4.4$ bash-4.4$ git pull remote: Enumerating objects: 7, done. remote: Counting objects: 100% (7/7), done. remote: Compressing objects: 100% (4/4), done. remote: Total 6 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (6/6), done. From /home/prog/0.4.1/remote/platform b5e76a7..6b831e4 master -> origin/master Updating <font color="#8b0000">b5e76a7..6b831e4</font> Fast-forward build-system/.gitrepo/config | 12 ++++++++++++ build-system/README | 3 +++ 2 files changed, 15 insertions(+) create mode 100644 build-system/.gitrepo/config create mode 100644 build-system/README bash-4.4$ 

the user will have at his disposal the code of the build-system subproject of exactly that revision, which was defined by the author of the project. The user can at any time refine the current revision using the config command:

 bash-4.4$ bash-4.4$ git subrepo config build-system/ commit Subrepo 'build-system' option 'commit' has value 'b2f507918f2821cb7dd90c33223ed5cc3c9922a2'. bash-4.4$ 

It is noteworthy that the user does not need to make additional settings and he can rely on the fact that the author of the project delivered to him exactly that build-system revision, which is necessary for the current version of the platform to work.

This is exactly what the author of the project wanted.

Delivery of changes to the upstream project

Suppose now that our user is a member of the project and is allowed to deliver changes not only to the remote / platform.git upstream repository, but also to the upstream repository of the remote / build-system.git subproject .

Then, if the user makes a change:

 bash-4.4$ bash-4.4$ cd build-system/ bash-4.4$ vim README bash-4.4$ cat README [master] build-system 1.0.1 bash-4.4$ bash-4.4$ git commit -a -m "update BS version to 1.0.1" [master d30b001] update BS version to 1.0.1 1 file changed, 1 insertion(+), 1 deletion(-) bash-4.4$ bash-4.4$ cd .. bash-4.4$ git log commit d30b001286b08708f5c30c1f5346a90e4339f969 (HEAD -> master) Author: user <___@_____> Date: Tue Oct 30 10:49:32 2018 +0300 update BS version to 1.0.1 . . . bash-4.4$ 

he can put them in the upstream repository as follows:

 bash-4.4$ bash-4.4$ git subrepo push build-system/ Subrepo 'build-system' pushed to '../../remote/build-system.git' (master). bash-4.4$ 

 bash-4.4$ bash-4.4$ git status On branch master Your branch is ahead of 'origin/master' by 2 commits. (use "git push" to publish your local commits) nothing to commit, working tree clean bash-4.4$ bash-4.4$ git push Enumerating objects: 14, done. Counting objects: 100% (14/14), done. Delta compression using up to 4 threads Compressing objects: 100% (7/7), done. Writing objects: 100% (9/9), 992 bytes | 992.00 KiB/s, done. Total 9 (delta 1), reused 0 (delta 0) To /home/prog/0.4.1/remote/platform.git d00be9f..deccb66 master -> master bash-4.4$ 

Direct work with the upstream repository

Now consider what happened in the remote / build-system.git upstream repository .

 bash-4.4$ bash-4.4$ cd owner/build-system/ bash-4.4$ bash-4.4$ git pull remote: Enumerating objects: 5, done. remote: Counting objects: 100% (5/5), done. remote: Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. From /home/prog/0.4.1/remote/build-system b2f5079..d229920 master -> origin/master Updating b2f5079..d229920 Fast-forward README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) bash-4.4$ bash-4.4$ git log commit d229920c7de34405bc7b8d47f36d420987687908 (HEAD -> master, origin/master) Author: user <___@_____> Date: Tue Oct 30 10:49:32 2018 +0300 update BS version to 1.0.1 commit b2f507918f2821cb7dd90c33223ed5cc3c9922a2 Author: user <___@_____> Date: Tue Oct 30 10:05:30 2018 +0300 init build-system master 1.0.0 bash-4.4$ 

That is, the author of the project received the changes made by the project participant.

Of course, the author can make changes directly to the upstream-repository of the build-system project:

 bash-4.4$ bash-4.4$ cd owner/build-system/ bash-4.4$ bash-4.4$ vim README bash-4.4$ cat README [master] build-system 1.0.2 bash-4.4$ git commit -a -m "update build-system version to 1.0.2" [master 8255f59] update build-system version to 1.0.2 1 file changed, 1 insertion(+), 1 deletion(-) bash-4.4$ git push Enumerating objects: 5, done. Counting objects: 100% (5/5), done. Writing objects: 100% (3/3), 281 bytes | 281.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To /home/prog/0.4.1/remote/build-system.git d229920..8255f59 master -> master bash-4.4$ 

And all participants, as well as users of the main project, can receive these changes using the git subrepo pull command

 bash-4.4$ bash-4.4$ cd owner/platform/ bash-4.4$ bash-4.4$ git subrepo pull build-system/ Subrepo 'build-system' pulled from '../../remote/build-system.git' (master). bash-4.4$ git status On branch master Your branch is ahead of 'origin/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working tree clean bash-4.4$ git push Enumerating objects: 11, done. Counting objects: 100% (11/11), done. Delta compression using up to 4 threads Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 670 bytes | 670.00 KiB/s, done. Total 6 (delta 1), reused 0 (delta 0) To /home/prog/0.4.1/remote/platform.git 6b831e4..b6f4a7b master -> master bash-4.4$ 

findings

If the developer does not need to keep the history of subprojects in the main repository and, when delivering the code, he operates with branches, rather than fixed tags, then git-subrepo is quite suitable for organizing daily work.

Conventionally, among the disadvantages of git-subrepo is the fact that git subrepo clone operation is possible only in relation to the subproject branches. In other words, the user cannot connect a subproject referring to his fixed tag or a specific revision, that is, commands like

 bash-4.4$ git subrepo clone ../../remote/build-system.git build-system -t 1.0.1 bash-4.4$ git subrepo clone ../../remote/build-system.git build-system 7f5d1113eb0bc6 

not valid.

LITERATURE:

• Git - Subtree Merging
• Mastering Git subtrees
• Mastering Git submodules
• Git subrepo
• Branch model and git module management for a large project