An example implementation of Continuous Integration using BuildBot

(Image by Computerizer from Pixabay)

Hello!

My name is Evgeny Cherkin , I am a programmer of the development team at the mining company Polymetal .

Starting any large project you start to think: “What software is better to use for its maintenance?”. IT-project before the release of the next version goes through a series of stages. Well, when the chain of these stages is automated. By itself, the automated process of releasing a new version of an IT project is called Continuous Integration . BuildBot for us was a good helper, implementing this process.
')
In this article, I decided to present an overview of the capabilities of BuildBot . What is this software capable of? How to approach it and how to build with it normal EFFECTIVE WORKING RELATIONS? You can also apply our experience in your own home by creating a working service for assembling and testing your project on your machine.

1. Why BuildBot?

---

Earlier on habr-e, I met articles on the implementation of Continuous Integration using BuildBot . For example, this one seemed to me the most informative. There is another example - easier . These articles can be seasoned with an example from the manual , and this will follow in English. The coupe is a good starting point. After reading these articles, you probably immediately want to do something on the BuildBot .

Stop! Did anyone even use it in their projects? It turns out yes, many have applied it in their tasks. You can find examples of using BuildBot in the archives of Google codes.

So what is the logic of people using Buildbot ? After all, there are other tools: CruiseControl and Jenkins . I will answer this way. For most tasks, Jenkins really will be enough. In turn, the BuildBot is more adaptive, while the tasks there are solved as easily as in Jenkins . Choose you. But since we are looking for a tool for a developing target project, then why not choose the one that allows, building on simple steps, to get an assembly system that has interactivity and a unique interface.

For those whose target project is written in python, the question arises: “Why not choose an integration system that has a clear interface from the point of view of the language used in the project?”. And then it's time to present the benefits of BuildBot .

So, our “instrumental quartet”. For myself, I defined the four features of BuildBot :

1. This is an open source framework under the GPL license.
2. This is the use of python as a tool for configuring and describing the required actions.
3. This is an opportunity to receive a response from the side of the machine on which the assembly takes place.
4. This, finally, the minimum requirements for the host. Deployment requires python and twisted, and does not require a virtual machine and java-machine.

2. Concept led by BuildMaster

---

A central place in the task distribution architecture is BuildMaster . It is a service that:

• tracks changes in the project source tree
• sends commands that Worker should execute to build a project and test it
• notifies users of the results of actions performed

BuildMaster is configured through the master.cfg file. This file is in the root of BuildMaster . Later I will show how this root is created. By itself, the master.cfg file contains python, a script that uses BuildBot calls.

The next most important BuildBot object is called Worker . This service can be run on another host from another OS, and maybe on where BuildMaster . It can also exist in a specially prepared virtual environment with its own packages and variables. These virtual environments can be prepared using python utilities like vertualenv, venv .

BuildMaster transmits commands to each Worker , and that, in turn, executes them. That is, it turns out that the process of building and testing a project can go on a Worker running Windows and on another Worker running linux.

The checkout of project source codes occurs at each Worker .

3. Installation

---

So let's go. As a host, I will use Ubuntu 18.04. On it, I will put one BuildMaster -a and one Worker -a. But first you need to install python3.7:

sudo apt-get update sudo apt-get install python3.7 

For those who need python3.7.2 instead of 3.7.1, you can do the following:

 sudo apt-get update sudo apt-get software-properties-common sudo add-apt-repository ppa:deadsnakes/ppa sudo apt-get install python3.7 sudo ln -fs /usr/bin/python3.7 /usr/bin/python3 pip3 install --upgrade pip 

The next step is to install Twited and BuildBot , as well as packages that allow you to use the additional functionality of BuildBot -a.

 /*   sudo        /usr/local/lib/python3.7/dist-packages*/ #     Worker- sudo pip install twisted # twisted sudo pip install buildbot #BuildMaster #  pip install pysqlite3 #  sqllite    pip install jinja2 #framework  django,  web     pip install autobahn #Web c   BuildMaster->Worker pip install sqlalchemy sqlalchemy-migrate #     # Web  BuildBot-a pip install buildbot-www buildbot-grid-view buildbot-console-view buildbot-waterfall-view pip install python-dateutil #   web #         pip install buildbot-worker #Worker #  sudo pip install virtualenv #  

4. First steps

---

Time to create a BuildMaster . We will have it in the folder / home / habr / master .

 mkdir master buildbot create-master master #     

Next step. Create a Worker . We will have it in the folder / home / habr / worker .

 mkdir worker buildbot-worker create-worker --umask=0o22 --keepalive=60 worker localhost:4000 yourWorkerName password 

When you start Worker , by default it will create a folder in / home / habr / worker with the name of the project that is specified in master.cfg . And in the daddy with the name of the project he will create a build directory, and then he will checkout there . The work directory for the Worker will be the / home / habr / yourProject / build directory.

"Golden Key
And now, for the sake of what I wrote the previous paragraph: the script that the Master will require the Worker to do remotely in this directory will not be executed, since the script does not have permissions to run. To remedy the situation, you need a key --umask = 0o22 , which imposes a ban on writing to this directory, but will leave the launch rights. And we just need it.

BuildMaster and Worker establish a connection between themselves. It happens that it breaks off and the Worker awaits a response from BuildMaster for a while. If no response is given, the connection is restarted. The key --keepalive = 60 is just needed in order to specify the time after which connect restarts.

5. Configuration. Step-by-step recipe

---

The BuildMaster configuration is maintained on the side of the machine, where we executed the create-master command. In our case, this is the directory / home / habr / master . The configuration file master.cfg does not yet exist, but the team itself has already created the file master.cmg.sample . You must rename it in master.cfg.sample in master.cfg

 mv master.cfg.sample master.cfg 

Open this master.cfg . And analyze what it consists of. And then try to make your configuration file.

5.1 BuildmasterConfig

 c = BuildmasterConfig = {} 

BuildmasterConfig - the basic dictionary of the configuration file. It must be necessarily involved in the configuration file. For ease of use in the configuration code is entered its alias "c" . Key names in c [“keyFromDist”] are fixed elements for interacting with BuildMaster . Under each key, the corresponding object is substituted as a value.

5.2 workers

 c['workers'] = [worker.Worker("example-worker", "pass")] 

This time we specify the BuildMaster list of Workers . We created the Worker above , specifying you-worker-name and password . Now they need to be specified instead of example-worker and pass .

5.3 change_source

 c['change_source'] = [] c['change_source'].append(changes.GitPoller( 'git://github.com/buildbot/hello-world.git', workdir='gitpoller-workdir', branch='master', pollInterval=300)) 

Using the change_source key of the c dictionary, we get access to the list where you want to put the object that polls the repository with the project source code. The example uses the Git repository, which is polled with a certain frequency.

The first argument is the path to your repository.

workdir is the path to the folder where, on the Worker side, relative to the path / home / habr / worker / yourProject / build, git will store the local version of the repository.

branch contains a specific branch in the repository, which should be monitored.

pollInterval contains the number of seconds after which the BuildMaster will poll the repository for changes.

There are several methods to track changes in the project repository.

The easiest method is Polling , which implies that BuildMaster periodically polls the server with the repository. If the commit reflected the changes in the repository, then BuildMaster with some delay will create an internal Change object and send it to the Scheduler event handler, which will launch the steps to build and test the project on the Worker . Among these steps will be specified update repository. It is on Worker that a local copy of the repository will be created. The details of this process will be explained below in the next two sections ( 5.4 and 5.5 ) .

An even more elegant method of tracking changes in the repository is direct sending messages from the server on which it is hosted to BuildMaster — about changing the source codes of a project. In this case, as soon as the developer makes a commit , the server with the project repository will send a message to BuildMaster . And that, in turn, will be intercepted by creating a PBChangeSource object. Further, this object will be transferred to Scheduler , which activates the steps for building the project and testing it. An important part of this method is working with server hooks in the repository. In the hook script, which is responsible for handling actions during commit , you must call the sendchange utility and specify the network address of BuildMaster . You need to specify the network port that the PBChangeSource will listen to. PBChangeSource , by the way, is part of BuildMaster . This method will require admin -a privileges on the server where the project repository is located. You must first make a backup repository.

5.4 shedulers

 c['schedulers'] = [] c['schedulers'].append(schedulers.SingleBranchScheduler( name="all", change_filter=util.ChangeFilter(branch='master'), treeStableTimer=None, builderNames=["runtests"])) c['schedulers'].append(schedulers.ForceScheduler( name="force", builderNames=["runtests"])) 

schedulers is an element that acts as a trigger that runs the entire chain of assembly and testing of the project.


Those changes that were committed by change_source were transformed during the work of the BuildBot -a into a Change object, and now each Sheduler builds requests for starting the project build process on their basis. However, it also determines when to send these requests further in the queue. An object Builder keeps a queue of requests and monitors the status of the current assembly on a separate Worker- e. Builder exists on both BuildMaster -e and Worker -e. He also sends from BuildMaster to Worker a specific build , a series of steps that should be performed.
We see that in the current example of such schedulers 2 pieces are created. Moreover, each has its own type.

SingleBranchScheduler is one of the most popular schedule classes. It observes one branch and is triggered by a fixed change in it. When he sees the changes, he can postpone sending the build request (postpone for the period specified in the treeStableTimer special parameter). The name specifies the name of the schedule that will be displayed in the BuildBot- web interface. In ChangeFilter, a filter is set, after passing which changes in the branch cause the schedule to send a build request. In builderNames , the name builder -a is specified, which we will specify later. The name in our case will be the same as the project name: yourProject .

ForceScheduler is a very simple thing. This type of schedule is triggered by a mouse click through the BuildBot -web interface. Parameters have the same essence as in SingleBranchScheduler .

PS number 3. Suddenly useful
Periodic is a schedule that is triggered at a certain fixed time interval. It looks like this

 from buildbot.plugins import schedulers nightly = schedulers.Periodic(name="daily", builderNames=["full-solaris"], periodicBuildTimer=24*60*60) c['schedulers'] = [nightly] 

5.5 BuildFactory

 factory = util.BuildFactory() factory.addStep(steps.Git(repourl='git://github.com/buildbot/hello-world.git', mode='incremental')) factory.addStep(steps.ShellCommand(command=["trial", "hello"], env={"PYTHONPATH": "."})) 

periodicBuildTimer sets the time of this periodicity in seconds.

BuildFactory creates a specific build , which the builder then sends to the Worker . The BuildFactory indicates the steps to be performed by the Worker . Steps are added by calling the addStep method .


The first step added in this example is git clean -d -f -f –x , then git checkout . These actions are included in the method parameter, which is not clearly indicated, but implies the default value of fresh . The parameter mode = 'incremental' says that the files from the directory where chechout is being made , while those that are missing from the repository, remain intact.

The second step added is to call the trial script with the hello parameter on the Worker side from the / home / habr / worker / yourProject / build directory with the PATHONPATH = environment variable ... ... Thus, you can write your own scripts and execute them on the Worker -a side through the util.ShellCommand step. These scripts can be put directly into the repository. Then when chechout -e they will fall into / home / habr / worker / yourProject / build . However, then there are two "but":

1. The Worker must be created with the --umask switch so that it does not block execution rights after checkout -a.
2. When git push these scripts, you must specify the exacutable property, so that when chechout -e you do not lose the right to execute the Git script.

5.6 builders

 c['builders'] = [] c['builders'].append(util.BuilderConfig(name="runtests", workernames=["example-worker"], factory=factory)) 

About what Builder was told here . Now I will tell you more about how to create it. BuilderConfig is a builder constructor. There are several such constructors in c ['builders'] , since this is a list of builder type objects. Now let's rewrite the example from BuildBot , bringing it closer to our task.

 c['builders'] = [] c['builders'].append(util.BuilderConfig(name="yourProject", workernames=["yourWorkerName"], factory=factory)) 

Now I will tell you about the parameters BuilderConfig .

name specifies the name of the builder -a. Here we called it yourProject . This means that on the Worker , this same path will be created / home / habr / worker / yourProject / build . Sheduler is looking for a builder by that name.

workernames contains a list of workers . Each of which must be added to c ['workers'] .

factory - a specific build with which the builder is associated. It will send the build object to the Worker to perform all the steps that make up this build -a.

6. Example of custom configuration

---

Here is the sample project architecture that I propose to implement via BuildBot
.

As a version control system, we will use svn . The repository itself will be located in a certain cloud. Here is the address of this cloud svn.host/svn/yourProject/trunk . In the cloud under svn there is a username: user account, passwd: password . Scripts that are build -a steps will also be located in the svn branch, in a separate buildbot / worker_linux folder . These scripts are in the repository with the executable property saved.

BuildMaster and Worker work on the same project.host host. BuildMaster stores its files in the / home / habr / master folder. Worker stores the following path / home / habr / worker . BuildMaster -a and Worker- processes are connected through the 4000th port using the BuildBot -a protocol, that is, the 'pb' protocol.

The target project is written entirely in python. The task is to track its changes, create an executable file, generate documentation, carry out testing. In case of failure, all developers need to send a message to the mail stating that there is a failed action.

Web mapping BuildBot we will connect to port 80 for project.host . Apatch is not required. The twisted library already contains a web server, BuildBot uses it.

To store the internal information for BuildBot, we will use sqlite .

For mailing, you need the smtp.your.domain host - it allows sending emails from projectHost@your.domain without authentication. Also, on the smtp host, the protocol is heard on post 1025.

There are two persons involved in the process: admin and user . admin admins BuildBot . user is the commit commit .

The exacutable file is generated via pyinstaller . Documentation is generated via doxygen .

For this architecture, I wrote this master.cfg :


First you need to create BuildMaster -a and Worker -a. Then paste this master.cfg file into / home / habr / master .

The next step is to start the BuildMaster service .

 sudo buildbot start /home/habr/master 

Then start the Worker -a service.

 buildbot-worker start /home/habr/worker 

Done! Now Buildbot will track changes and work on a commit in svn , following the steps of building and testing a project with the above architecture.

Below I will write down some of the features of the above master.cfg.

6.1 On the way to my master.cfg

A lot of mistakes will be made while writing your master.cfg , so you will need to read the log file. It is stored both on BuildMaster -ec in absolute way /home/habr/master/twistd.log , and on the Worker -a side with absolute way /home/habr/worker/twistd.log . As you read the error and fix it, you will need to restart the BuildMaster -a service. Here is how it is done:

 sudo buildbot stop /home/habr/master sudo buildbot upgrade-master /home/habr/master sudo buildbot start /home/habr/master 

6.2 Working with svn

 svn_poller = changes.SVNPoller(repourl="https://svn.host/svn/yourProject/trunk", svnuser="user", svnpasswd="password", pollinterval=60, split_file=util.svn.split_file_alwaystrunk ) c['change_source'] = svn_poller hourlyscheduler = schedulers.SingleBranchScheduler( name="your-project-schedulers", change_filter=util.ChangeFilter(branch=None), builderNames=["yourProject"], properties = {'owner': 'admin'} ) c['schedulers'] = [hourlyscheduler] checkout = steps.SVN(repourl='https://svn.host/svn/yourProject/trunk', mode='full', method='fresh', username="user", password="password", haltOnFailure=True) 

First, take a look at svn_poller . This is all the same interface that regularly polls the repository once a minute. In this case, svn_poller refers only to the trunk branch. The mysterious parameter split_file = util.svn.split_file_alwaystrunk sets the rules: how to split the svn folder structure into branches. He offers them relative paths. In turn, split_file_alwaystrunk simplifies the process by saying that only trunk is in the repository.

In Schedulers , a ChangeFilter is specified , which sees None and associates with it the trunk branch by the specified association through split_file_alwaystrunk . In response to changes in the trunk , it starts the builder with the name yourProject .

The properties here are needed in order for the admin to receive a mailing from the results of the assembly and testing as the owner of the process.

The build -a checkout step is capable of completely deleting any files in the local version of the Worker repository. A then do a full svn update . The mode is configured through the parameter mode = full , method = fresh . The haltOnTailure parameter says that if svn update is executed with an error, then the entire build and testing process should be suspended, since further actions do not make sense.

6.3 Letter to you: reporters authorized to declare

reporters is a mailing notification service.

 template_html=u'''\ <h4>  : {{ summary }}</h4> <p>   : {{ workername }}</p> <p>: {{ projects }}</p> <p>         : {{ buildbot_url }}</p> <p>         : {{ build_url }}</p> <p> WinSCP     c ip:xxx.xx.xxx.xx.   habr/password,   executable    ~/worker/yourProject/build/dist.</p> <p><b>    Buildbot</b></p> ''' sendMessageToAll = reporters.MailNotifier(fromaddr="projectHost@your.domain", sendToInterestedUsers=True, lookup="your.domain", relayhost="smtp.your.domain", smtpPort=1025, mode="warnings", extraRecipients=['user@your.domain'], messageFormatter=reporters.MessageFormatter( template=template_html, template_type='html', wantProperties=True, wantSteps=True) ) c['services'] = [sendMessageToAll] 

He can send messages in different ways .

MailNotifier uses mail to send notifications.

template_html specifies the template text to be distributed. To create markup is used html. It is modified by the jinja2 engine (can be compared with django ). BuildBot has a set of variables, the values ​​of which are substituted into the template during the formation of the message text. These variables are inscribed in {{double braces}}. For example, summary displays the status of completed operations, that is, success or failure. And the projects will display yourProject . So, with the help of control commands in jinja2 , BuildBot variables and python string formatting tools, you can create a completely informative message.

MailNotifier contains the following arguments.

fromaddr - address from which all will receive mailing.

sendToInterestedUsers = True sends a message to the owner and the user who made the commit .

lookup is the suffix to add to the names of users who receive the feed. So admin as the user will receive the newsletter at admin@your.domain.

relayhost sets the name of the host on which the smtp server is open, and smptPort sets the port number that the smtp server listens to.

mode = "warning" says that the distribution should be done only if there is at least one build step , which ended with the status of failure or warning. In the case of success, the mailing list is not required.

extraRecipients contains a list of persons to whom the mailing list should be sent, in addition to the owner and the person who made the commit .

messageFormatter is an object that defines the message format, its template, and the set of variables available from jinja2 . Parameters such as wantProperties = True and wantSteps = True set this set of available variables.

with ['services'] = [sendMessageToAll] provides a list of services, among which will be ourreporter .

We did it! My congratulations

---

We created our own configuration and saw the functionality that BuildBot is capable of . This, I think, is enough to understand whether this tool is needed to create your project. Is he interesting to you? Will it be useful to you? Is it convenient to work with him? Then I write this article for good reason.

And further.It would be great if the professional community using the BuildBot becomes wider, the manuals are translated, and there are more examples.

Thank you all for your attention. Good luck.