Become a manager. Part four
While fans of the exotic in Habré are actively drinking cups with Java, taking F # doses and injecting themselves other Haskell, we learned to collect their creations in deb-packages. Since the previous part of the time, someone has probably already accumulated several ready-made packages, and we have not even tried to put them into the official repository of Debian and Ubuntu yet. Therefore, it is time to think about how to organize all the accumulated wealth into one big beautiful repository that other users will not be ashamed to offer.
(Parts 1 , 2 and 3 )
For these purposes, we will consider two programs (simply because to comprehend the dao I had them) - this is apt-ftparchive from the standard apt-utils package and the reprepro package.
Let's first take a look at how a normal repository works. Repositories are of two types - ugly and oblique (or in the original - flat - flat, we will not consider them) and normal hierarchical ones. Here is the hierarchical we will learn to create. So, a normal repository consists of two directories: pool and dists .
pool /
In this directory in an arbitrary hierarchy may lie packages. I say “in an arbitrary” because you can really lay them out as you please, but in practice the following hierarchy is usually implemented:
Here “main” is the name of the section to which we assign our package (see sections in the first article), then comes the first letter of the name of the src package and then the directory named after the src package. This directory will contain the src-package and all binary packages collected from it. For numerous libraries where all package names start with “lib”, instead of the letter “l”, a separate “liba” is used, where “a” is the first letter of the name after “lib”.
')
dists /
This directory is much more complicated. First of all, we have to find a directory with the name of the repository in it - in ubuntu it is various “intrepid”, “intrepid-proposed”, “jaunty”, etc., in debian it is “etch”, “etch-proposed-updates” , “Lenny” and their synonyms - “stable”, “testing”, etc. - as a rule, they are symlinks to the canonical name. Thus, you can store packages in one place at once for several repositories.
dists / unstable / [1]
Inside this directory you can find the following files:
In addition, in the same directory we will find directories with names corresponding to the names of repository sections - main, contrib, non-free or main, universe, multiverse, restricted or those that you define yourself.
dists / unstable / main /
In each section (not only main, of course) you can observe a set of directories of the following form:
dists / unstable / main / binary-i386 /
The Packages file is stored here (as well as Packages.gz and Packages.bz2 ), which contains descriptions of the binary packages of the corresponding architecture. In fact, it is a combination of control-files from all binary packages of the repository for this architecture. Often, here you can still find the Release file, which, unlike the same file in dists / unstable /, does not contain file hashes, but stores only repository descriptions for a given directory.
dists / unstable / main / source /
Here everything is the same as in binary-arch, except that the word Sources is used instead of Packages.
Of course, all these files can be written with pens, but I would not advise you. Especially if you have at least 3-4 packages. Therefore, we will use the magic utility apt-ftparchive from the apt-utils package and dpkg-scanpackages from the dpkg-dev package.
So, let's say we have a certain qurm src-package created by us in the previous series, and binary packages qutim and qutim-dev collected from it. We collected them under two architectures, so now we have the following set of files:
The first three files, as we remember, are the src-package, and the rest are the collected binary packages. We also have the qutim_0.1.99.138-1_i386.changes and qutim_0.1.99.138-1_amd64.changes files generated during the package build.
Let's put together a minimally decent repository.
So, create a directory with the initial structure and copy the packages into it:
As you can see, the placement of the packages themselves in the repository is nothing at all complicated. It will be a little harder to generate a set of descriptions for these packages:
Here we generate descriptions for each architecture and for src packages, after which we put them in the appropriate file. Notice that we generate three files each — Packages , Packages.gz, and Packages.bz2 (similar to Source). In principle, this is not necessary, but we will be cultured people and will generate all three - choose any one you like. The -a option specifies which packages for which architecture to select from the entire set in pool / main. As I said before, the qutim-dev package will hit
Theoretically, the generation of the description of binary packages could have been done with apt-ftparchive, but I still did not find in it the options that allocate packets of a certain architecture instead of pushing everything into the output.
Now we need to generate the release file:
The description still has to be written by hand, apt-ftparchive can not do it.
And the last step is to sign our repository:
Voila, our repository can be used. Add the following lines to /etc/apt/sources.list:
We update the list of packages - and the repository can be used.
Of course, this is not very convenient - you need to regenerate the description files every time you update a set of packages. Of course, all this is automated by scripts, but still so uninteresting. Therefore, we proceed to the second program used, which will do everything for us.
This program is just a paradise for those who do not want to bother with the structure of the repository at all, but want to configure and use once. Therefore, we demolish our homegrown repository and install the reprepro package:
Now we create a distributions file in the rep / conf directory, in which we will describe our repository:
Codename should contain the repository name (lenny, hardy, etch, jaunty), and Suite - a synonym for creating symbolic links (stable, unstable). The Version, Origin, Label and Description parameters do not carry any special meaning and are simply copied to the Release file. The SignWith parameter indicates the need to sign the repository by creating the file Release.gpg. DebIndices, DscIndices, and Contents — indicate the need to create Packages, Sources, and Contents-arch files in a simple and archived form.
Now we can add our packages:
the -b switch indicates repro where our repository is located, the --ask-passphrase command reports that our gpg-key is password-protected and when signing the distribution kit it would be nice to ask for the password, the -C switch reports that we include the package in the main section. One interesting point should be noted here. The include command takes as an argument the name of the repository in which to include the file and the name of the .changes file created when building the src package. At the same time, the src-package and the collected binary packages for this architecture are added to the repository. And here we have a problem - the reprepro developers flatly refuse to add packages of an already existing version to the repository. Therefore, if we now try to specify the .changes file for a different architecture, we will get an error - we have already added not only the src package, but also the qutim-dev package, compiled for all architectures. For this case, there is an includedeb command that allows you to specify only a deb-file with a different architecture.
Now browse the contents of the rep directory. The conf subdirectory contains the configuration of the repository (more about it can be found in ` man reprepro` ), the db directory is its internal database. But the pool and dists directories became very different from what we created with apt-ftparchive and became similar to the structure I described at the beginning. Since we already have the repository address in sources.list, type ` sudo apt-get update` and use it :)
And finally, if you want to study the structure of the repository in more detail, I recommend studying the official Ubuntu repository . Those who wish, of course, may also familiarize themselves with the Debian repository , but I immediately warn you that it is much more complex.
* [1] here and hereinafter, instead of the abstract catalog name, I will use concrete examples
(Parts 1 , 2 and 3 )
For these purposes, we will consider two programs (simply because to comprehend the dao I had them) - this is apt-ftparchive from the standard apt-utils package and the reprepro package.
Repository structure
Let's first take a look at how a normal repository works. Repositories are of two types - ugly and oblique (or in the original - flat - flat, we will not consider them) and normal hierarchical ones. Here is the hierarchical we will learn to create. So, a normal repository consists of two directories: pool and dists .
pool /
In this directory in an arbitrary hierarchy may lie packages. I say “in an arbitrary” because you can really lay them out as you please, but in practice the following hierarchy is usually implemented:
pool / main / q / qutim /
Here “main” is the name of the section to which we assign our package (see sections in the first article), then comes the first letter of the name of the src package and then the directory named after the src package. This directory will contain the src-package and all binary packages collected from it. For numerous libraries where all package names start with “lib”, instead of the letter “l”, a separate “liba” is used, where “a” is the first letter of the name after “lib”.
')
dists /
This directory is much more complicated. First of all, we have to find a directory with the name of the repository in it - in ubuntu it is various “intrepid”, “intrepid-proposed”, “jaunty”, etc., in debian it is “etch”, “etch-proposed-updates” , “Lenny” and their synonyms - “stable”, “testing”, etc. - as a rule, they are symlinks to the canonical name. Thus, you can store packages in one place at once for several repositories.
dists / unstable / [1]
Inside this directory you can find the following files:
- Contents-arch is a file containing a list of all files from all packages. Instead of “arch”, the name of the package architecture is substituted, and at the end there may still be “.gz” and “.bz2” (if the file is compressed accordingly). It is optional and is used mainly by utilities of the apt-file type to search in which package the desired file can be found. Example name - Contents-amd64.gz
- Release is a file containing a brief description of the repository and hashes of all auxiliary files contained in dists / .
- Release.gpg is the gpg-signature of the Release file, certifying the creation of a file by a specific author. Both files are optional, but if they are not available, APT is guaranteed to swear at an unauthenticated repository.
In addition, in the same directory we will find directories with names corresponding to the names of repository sections - main, contrib, non-free or main, universe, multiverse, restricted or those that you define yourself.
dists / unstable / main /
In each section (not only main, of course) you can observe a set of directories of the following form:
- binary-arch is a directory for describing packages of the corresponding architecture, for example, binary-amd64 is for the amd64 architecture. All architecture packages (such as dev packages) will be listed in the directory for each architecture.
- source - src-packages are described in this directory.
- i18n - this directory is rarely used and mostly only in official repositories - it contains files with translation of package descriptions into other languages. I will not describe this directory, because it is not such a really important element, if you want to use it - you can always look at the ubuntu repository
dists / unstable / main / binary-i386 /
The Packages file is stored here (as well as Packages.gz and Packages.bz2 ), which contains descriptions of the binary packages of the corresponding architecture. In fact, it is a combination of control-files from all binary packages of the repository for this architecture. Often, here you can still find the Release file, which, unlike the same file in dists / unstable /, does not contain file hashes, but stores only repository descriptions for a given directory.
dists / unstable / main / source /
Here everything is the same as in binary-arch, except that the word Sources is used instead of Packages.
apt-ftparchive
Of course, all these files can be written with pens, but I would not advise you. Especially if you have at least 3-4 packages. Therefore, we will use the magic utility apt-ftparchive from the apt-utils package and dpkg-scanpackages from the dpkg-dev package.
So, let's say we have a certain qurm src-package created by us in the previous series, and binary packages qutim and qutim-dev collected from it. We collected them under two architectures, so now we have the following set of files:
qutim_0.1.99.138.orig.tar.gz
qutim_0.1.99.138-1.diff.gz
qutim_0.1.99.138-1.dsc
qutim_0.1.99.138-1_amd64.deb
qutim_0.1.99.138-1_i386.deb
qutim-dev_0.1.99.138-1_all.debThe first three files, as we remember, are the src-package, and the rest are the collected binary packages. We also have the qutim_0.1.99.138-1_i386.changes and qutim_0.1.99.138-1_amd64.changes files generated during the package build.
Let's put together a minimally decent repository.
So, create a directory with the initial structure and copy the packages into it:
$ mkdir -p rep/dists
$ mkdir -p rep/pool/main
$ cp qutim_0.1.99.138.orig.tar.gz qutim_0.1.99.138-1.diff.gz qutim_0.1.99.138-1.dsc qutim_0.1.99.138-1_amd64.deb qutim_0.1.99.138-1_i386.deb qutim-dev_0.1.99.138-1_all.deb rep/pool/main/
$ cd repAs you can see, the placement of the packages themselves in the repository is nothing at all complicated. It will be a little harder to generate a set of descriptions for these packages:
$ mkdir -p dists/unstable/main/binary-amd64
$ mkdir dists/unstable/main/binary-i386
$ mkdir dists/unstable/main/source
$ dpkg-scanpackages -a amd64 pool/main >dists/unstable/main/binary-amd64/Packages 2>/dev/null
$ dpkg-scanpackages -a amd64 pool/main | gzip -c9 >dists/unstable/main/binary-amd64/Packages.gz 2>/dev/null
$ dpkg-scanpackages -a amd64 pool/main | bzip2 -c9 >dists/unstable/main/binary-amd64/Packages.bz2 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main >dists/unstable/main/binary-i386/Packages 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main | gzip -c9 > dists/unstable/main/binary-i386/Packages.gz 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main | bzip2 -c9 > dists/unstable/main/binary-i386/Packages.bz2 2>/dev/null
$ apt-ftparchive sources pool > dists/unstable/main/source/Sources 2>/dev/null
$ apt-ftparchive sources pool | gzip -c9 > dists/unstable/main/source/Sources.gz 2>/dev/null
$ apt-ftparchive sources pool | bzip2 -c9 > dists/unstable/main/source/Sources.bz2 2>/dev/nullHere we generate descriptions for each architecture and for src packages, after which we put them in the appropriate file. Notice that we generate three files each — Packages , Packages.gz, and Packages.bz2 (similar to Source). In principle, this is not necessary, but we will be cultured people and will generate all three - choose any one you like. The -a option specifies which packages for which architecture to select from the entire set in pool / main. As I said before, the qutim-dev package will hit
Theoretically, the generation of the description of binary packages could have been done with apt-ftparchive, but I still did not find in it the options that allocate packets of a certain architecture instead of pushing everything into the output.
Now we need to generate the release file:
$ echo "Archive: unstable" > dists/unstable/Release
$ echo "Suite: unstable" >> dists/unstable/Release
$ echo "Components: main" >> dists/unstable/Release
$ echo "Origin: qutim.org" >> dists/unstable/Release
$ echo "Label: qutim.org Debian repository" >> dists/unstable/Release
$ echo "Architectures: amd64 i386" >> dists/unstable/Release
$ echo "Description: Debian qutIM unstable" >> dists/unstable/Release
$ apt-ftparchive release dists/unstable >> dists/unstable/ReleaseThe description still has to be written by hand, apt-ftparchive can not do it.
And the last step is to sign our repository:
$ gpg -abs -o dists/unstable/Release.gpg dists/unstable/ReleaseVoila, our repository can be used. Add the following lines to /etc/apt/sources.list:
deb file: /// path / to / rep unstable main
deb-src file: /// path / to / rep unstable main
We update the list of packages - and the repository can be used.
Of course, this is not very convenient - you need to regenerate the description files every time you update a set of packages. Of course, all this is automated by scripts, but still so uninteresting. Therefore, we proceed to the second program used, which will do everything for us.
reprepro
This program is just a paradise for those who do not want to bother with the structure of the repository at all, but want to configure and use once. Therefore, we demolish our homegrown repository and install the reprepro package:
$ cd .. && rm -rf rep/*
$ sudo apt-get install reprepro
$ mkdir rep/confNow we create a distributions file in the rep / conf directory, in which we will describe our repository:
Codename: lenny
Suite: unstable
Version: unstable
Origin: qutim.org
Label: qutim.org Debian Repository
Description: qutim.org Debian repository
Architectures: source i386 amd64
Components: main
SignWith: default
DebIndices: Packages Release. .gz .bz2
DscIndices: Sources Release. .gz .bz2
Contents:. .gz .bz2
Codename should contain the repository name (lenny, hardy, etch, jaunty), and Suite - a synonym for creating symbolic links (stable, unstable). The Version, Origin, Label and Description parameters do not carry any special meaning and are simply copied to the Release file. The SignWith parameter indicates the need to sign the repository by creating the file Release.gpg. DebIndices, DscIndices, and Contents — indicate the need to create Packages, Sources, and Contents-arch files in a simple and archived form.
Now we can add our packages:
$ reprepro -b rep/ createsymlinks
$ reprepro -b rep/ --ask-passphrase -C main include unstable qutim_0.1.99.138-1_amd64.changes
$ reprepro -b rep/ --ask-passphrase -C main includedeb unstable qutim_0.1.99.138-1_i386.debthe -b switch indicates repro where our repository is located, the --ask-passphrase command reports that our gpg-key is password-protected and when signing the distribution kit it would be nice to ask for the password, the -C switch reports that we include the package in the main section. One interesting point should be noted here. The include command takes as an argument the name of the repository in which to include the file and the name of the .changes file created when building the src package. At the same time, the src-package and the collected binary packages for this architecture are added to the repository. And here we have a problem - the reprepro developers flatly refuse to add packages of an already existing version to the repository. Therefore, if we now try to specify the .changes file for a different architecture, we will get an error - we have already added not only the src package, but also the qutim-dev package, compiled for all architectures. For this case, there is an includedeb command that allows you to specify only a deb-file with a different architecture.
Now browse the contents of the rep directory. The conf subdirectory contains the configuration of the repository (more about it can be found in ` man reprepro` ), the db directory is its internal database. But the pool and dists directories became very different from what we created with apt-ftparchive and became similar to the structure I described at the beginning. Since we already have the repository address in sources.list, type ` sudo apt-get update` and use it :)
And finally, if you want to study the structure of the repository in more detail, I recommend studying the official Ubuntu repository . Those who wish, of course, may also familiarize themselves with the Debian repository , but I immediately warn you that it is much more complex.
* [1] here and hereinafter, instead of the abstract catalog name, I will use concrete examples
Source: https://habr.com/ru/post/51427/
All Articles