Aptly - create your own repository
My organization is writing software under Linux.
The software is designed to work at retail outlets that are geographically distributed.
Initially, the software was provided to clients as a set of deb packages for different distributions and architectures with a list of packages that need to be installed as dependencies before installing these deb files.
I want to tell you what was the path of evolution from distributing files via ftp to creating a repository and launching a configuration management system and starting implementation of a continuous integration server.
So, software is a commercial development that is written in Qt, everything is built for two architectures (i386 and amd64) and for several distributions. At the moment, the following are defined: the last two or three Debian releases and the last two LTS from Ubuntu. Plus, there are several versions (currently three) that are used by customers.
Three versions turned out this way: when buying software, the client is offered support and, for quite reasonable money, providing new versions of software. Changing minor versions is free of charge, regardless of the availability of support. Major - or in the presence of support, or sold again only at a discount.
While there were few versions, and the number of installations by customers was small, it was quite possible to manage with ftp. It looked like this: after assembling all source codes in deb packages, a set of files for each client (depending on version) was archived and laid out for each client on ftp in its own section. Over time, the ftp accumulated a mass of identical tar-s of different clients, which had to be removed from time to time.
As time went on, customers were added, customer networks grew in size, and updating the software version on the points was another task, especially considering that 99% of the Internet points were (and still are) GPRS.
By the way, about collecting packages. Skipty, which collected deb packages, were originally written “300 years ago” on some manual in English and IMHO is not entirely correct. The following articles have helped to brush and wash the scripts (for which I want to say thanks to their authors):
Needless to say, the question was raised about distributing the collected packages in a more appropriate way. “And then Ostap suffered” (c), decided to do everything correctly, so that minimal time was spent on routine operations.
')
The reprepro is the software that supported the repository. At that time, the opportunities he provided were quite enough.
Further, based on the experience of updating, configuring a large number of computers, I wanted to have a configuration management tool. Google and Habr were called to help. Acquainted with ansible, chef, puppet and other systems, the names of which I do not remember. By understandability of configs, documentation, threshold of entry, and a combination of other parameters, the puppet was taken into service. The foreman was bolted to the puppet. And happiness has come.
Not much time passed and I wanted something more different as a control of the repository. Reprepro suited everyone except being able to store different versions of the same package in the same repository: to be able to quickly roll back to a specific version in the event of some kind of regression, or to quickly put the required version and test the behavior of a particular version on a specific data set on certain hardware configurations.
As an example: one client detected an application crash while working with documents. Their data set on developers' computers flew without questions, everything was opened and without dropping it worked remarkably well. We assembled a stand, installed the same linux, the same software version, again everything works and does not fall. They began to arrive as paranoidly as possible - they collected another machine as close as possible to the client’s machine, disk, memory, processor, monitor resolution, and lo and behold! repeated the fall. Began to understand: found that the essence of the monitor resolution, or rather not in the resolution as such, but in the ratio of the parties. The developers and the first test machine the ratio of the monitor was 16: 9, and the second test 4: 3. As a result, calculate that one line in qss results in a drop. That was, of course, a big surprise. So this is what this example is for. At the time of these tests, there was no such repository and on each assembled machine I had to copy deb packages in the old fashioned way through scp, install dependencies with pens, then install the packages themselves and then check. Instead of installing in a couple of minutes via aptitude with all dependencies.
I had to google. As a result, the article was nagged. Began to deal with the options described in it. I managed to look at:
The aptly documentation is quite detailed with examples. In addition, there is a bash-completion.
The possibilities of aptly are very broad, below are only those commands that I use. Those interested in more deeply recommend the product site .
You can create at once with all the parameters:
You can add the parameters later, and create just like this:
Change the parameters one by one:
Or all at once:
general information:
information with a list of packages:
single file:
whole catalog:
when adding, the lack of autocompletion of the file / directory path was somewhat embarrassing.
There is also support for snapshot-s, creating mirrors, moving packages between repositories, dependency support, built-in HTTP server for working with published repositories.
There is one feature about which the author wrote - pool is one common to all repositories, which is correct, only if I have files with the same name in different repositories, but different in content and size when publishing these repositories there are no messages, the logic should there was at least something to write. And in the pool, the first published repository file remains.
By the way, the author responded very quickly and created a task on the bug tracker .
At the moment, packages are collected by handles on the vitrual machines. What is not good. Work has already begun on the implementation of Jenkins . From the entire list of available funds for CI, he was chosen. TeamCity was originally installed, but the cost of licenses was not understood due to the fact that there are other projects that are completely free and have not the worst functionality. At least it looks like that now. If, during the operation of Jenkins, its capabilities for some reason will not be enough (although with so many plugins, this is hard to believe), we will change it for something better.
At the end of the article I would like to summarize.
The idea of ​​the article was to describe a wonderful tool - aptly, because in the open spaces Habra did not find anything about him. Well, to tell in the style of “here's a tool - you can use” was considered not interesting. I decided that it would be much more interesting to read which way my organization developed and which tools I used and to draw the community’s attention to a useful tool with the help of a personal example.
Thanks for attention.
The software is designed to work at retail outlets that are geographically distributed.
Initially, the software was provided to clients as a set of deb packages for different distributions and architectures with a list of packages that need to be installed as dependencies before installing these deb files.
I want to tell you what was the path of evolution from distributing files via ftp to creating a repository and launching a configuration management system and starting implementation of a continuous integration server.
Origin of life
So, software is a commercial development that is written in Qt, everything is built for two architectures (i386 and amd64) and for several distributions. At the moment, the following are defined: the last two or three Debian releases and the last two LTS from Ubuntu. Plus, there are several versions (currently three) that are used by customers.
Three versions turned out this way: when buying software, the client is offered support and, for quite reasonable money, providing new versions of software. Changing minor versions is free of charge, regardless of the availability of support. Major - or in the presence of support, or sold again only at a discount.
While there were few versions, and the number of installations by customers was small, it was quite possible to manage with ftp. It looked like this: after assembling all source codes in deb packages, a set of files for each client (depending on version) was archived and laid out for each client on ftp in its own section. Over time, the ftp accumulated a mass of identical tar-s of different clients, which had to be removed from time to time.
As time went on, customers were added, customer networks grew in size, and updating the software version on the points was another task, especially considering that 99% of the Internet points were (and still are) GPRS.
By the way, about collecting packages. Skipty, which collected deb packages, were originally written “300 years ago” on some manual in English and IMHO is not entirely correct. The following articles have helped to brush and wash the scripts (for which I want to say thanks to their authors):
Beginning of evolution
Needless to say, the question was raised about distributing the collected packages in a more appropriate way. “And then Ostap suffered” (c), decided to do everything correctly, so that minimal time was spent on routine operations.
')
The reprepro is the software that supported the repository. At that time, the opportunities he provided were quite enough.
Further, based on the experience of updating, configuring a large number of computers, I wanted to have a configuration management tool. Google and Habr were called to help. Acquainted with ansible, chef, puppet and other systems, the names of which I do not remember. By understandability of configs, documentation, threshold of entry, and a combination of other parameters, the puppet was taken into service. The foreman was bolted to the puppet. And happiness has come.
Not much time passed and I wanted something more different as a control of the repository. Reprepro suited everyone except being able to store different versions of the same package in the same repository: to be able to quickly roll back to a specific version in the event of some kind of regression, or to quickly put the required version and test the behavior of a particular version on a specific data set on certain hardware configurations.
As an example: one client detected an application crash while working with documents. Their data set on developers' computers flew without questions, everything was opened and without dropping it worked remarkably well. We assembled a stand, installed the same linux, the same software version, again everything works and does not fall. They began to arrive as paranoidly as possible - they collected another machine as close as possible to the client’s machine, disk, memory, processor, monitor resolution, and lo and behold! repeated the fall. Began to understand: found that the essence of the monitor resolution, or rather not in the resolution as such, but in the ratio of the parties. The developers and the first test machine the ratio of the monitor was 16: 9, and the second test 4: 3. As a result, calculate that one line in qss results in a drop. That was, of course, a big surprise. So this is what this example is for. At the time of these tests, there was no such repository and on each assembled machine I had to copy deb packages in the old fashioned way through scp, install dependencies with pens, then install the packages themselves and then check. Instead of installing in a couple of minutes via aptitude with all dependencies.
I had to google. As a result, the article was nagged. Began to deal with the options described in it. I managed to look at:
- DAK - did not master at all. I started it from her because “official solution”, but to my shame I couldn’t understand what to do with the project tree that git pulled me. Therefore, I didn’t strain myself and continued to read further.
- The second product that I noticed was mini-dak. And as if everything is clear from the configuration and the description is more or less clear. I just stopped the impossibility to make the names of the repositories (what is called suite in terms of mini-dak) such as I want. For example, I want to get the repository path like this:
deb http: // SERVER_NAME / squeeze-evolution-beta non-free
But I can't, because somewhere in the depth of the scripts a name of the variable with the signs “-” is created that does not work in bash-e. Attempts to modify the scripts with a file did not succeed, because my bash programming level somewhat does not reach the level of the person who wrote them, and did not want to kill a lot of time to learn all the subtleties of bash (though not the worst pastime). - The third option for review was chosen aptly. And it seems that this tool is suitable.
Work with aptly
The aptly documentation is quite detailed with examples. In addition, there is a bash-completion.
The possibilities of aptly are very broad, below are only those commands that I use. Those interested in more deeply recommend the product site .
Creating a repository:
You can create at once with all the parameters:
aptly repo create -comment="Wheezy prehistoric" -distribution="wheezy-prehistoric" -architectures="i386,amd64,all" -component="non-free" wheezy-prehistoric You can add the parameters later, and create just like this:
aptly repo create wheezy-prehistoric Change settings
Change the parameters one by one:
aptly repo edit -comment="Wheezy prehistoric" wheezy-prehistoric Or all at once:
aptly repo edit -comment="Wheezy prehistoric" -distribution="wheezy-prehistoric" -architectures="i386,amd64,all" -component="non-free" wheezy-prehistoric View existing repositories
aptly list Getting repository information
general information:
aptly show wheezy-prehistoric information with a list of packages:
aptly repo show -with-packages wheezy-prehistoric Adding packages to the repository
single file:
aptly repo add wheezy-prehistoric build//Debian-wheezy/chameleon-core_1.3.0-wheezy46_amd64.deb whole catalog:
aptly repo add wheezy-prehistoric build//Debian-wheezy when adding, the lack of autocompletion of the file / directory path was somewhat embarrassing.
Repository publication
aptly publish repo wheezy-prehistoric Deleting a post
aptly publish drop wheezy-prehistoric Get repository graph
aptly graph There is also support for snapshot-s, creating mirrors, moving packages between repositories, dependency support, built-in HTTP server for working with published repositories.
There is one feature about which the author wrote - pool is one common to all repositories, which is correct, only if I have files with the same name in different repositories, but different in content and size when publishing these repositories there are no messages, the logic should there was at least something to write. And in the pool, the first published repository file remains.
By the way, the author responded very quickly and created a task on the bug tracker .
Evolution continues
At the moment, packages are collected by handles on the vitrual machines. What is not good. Work has already begun on the implementation of Jenkins . From the entire list of available funds for CI, he was chosen. TeamCity was originally installed, but the cost of licenses was not understood due to the fact that there are other projects that are completely free and have not the worst functionality. At least it looks like that now. If, during the operation of Jenkins, its capabilities for some reason will not be enough (although with so many plugins, this is hard to believe), we will change it for something better.
PS
At the end of the article I would like to summarize.
The idea of ​​the article was to describe a wonderful tool - aptly, because in the open spaces Habra did not find anything about him. Well, to tell in the style of “here's a tool - you can use” was considered not interesting. I decided that it would be much more interesting to read which way my organization developed and which tools I used and to draw the community’s attention to a useful tool with the help of a personal example.
Thanks for attention.
Source: https://habr.com/ru/post/225825/
All Articles