Switching from CruiseControl.NET to Jenkins in the PVS-Studio development team

Now it is difficult to imagine software development without automated project builds and testing. To minimize the time spent on integrating developer changes into the project, there are various ready-made solutions. In this article I will talk about replacing the continuous integration server CruiseControl.NET with Jenkins in the PVS-Studio development team. And also about what prompted us to this, what goals we pursued and what problems we faced.

Introduction

Continuous Integration (eng. Continuous Integration, hereinafter CI) is an automated process for assembling, deploying and testing the software being developed. This development practice is popular both in large teams and among individual developers. For this practice, there are many ready-made solutions. This article will discuss free open source projects CruiseControl.NET and Jenkins.

CruiseControl.NET (CCNet) is a tool for continuous software integration implemented on the .NET Framework. There are also versions of the Java tool (CruiseControl) and version for Ruby environments (CruiseControl.rb). Manage and view information about the assembly is carried out through a web interface or desktop utility. Integrates with various version control systems. It is an open source project and, unfortunately, has not been developing since about 2013.

Jenkins is a continuous, open source integration tool written in Java. Was branched from the Hudson project after a dispute with Oracle. Providing continuous integration functions, it allows you to automate part of the software development process in which human participation is not necessary. Jenkins features can be expanded with plugins. At the moment, the project is actively developing and supported by both developers and the community.
')
Although the article is a bit similar to the review in the style “CCNet Vs. Jenkins ”, the emphasis will immediately be placed on choosing a server with Jenkins. The main reason for the change of the continuous integration tool for us is the lack of development of the project CruiseControl.NET. The article will also describe other aspects of working with CCNet, which had difficulties.

Recently, the PVS-Studio project had a 10-year anniversary, which can be found in the article “ How the PVS-Studio project began 10 years ago ”. For more than half the product's lifetime, we used CCNet. Its interface, settings and functions have become so familiar that Jenkins seemed extremely uncomfortable. We first began to use it when PVS-Studio for Linux appeared. The transition to Jenkins was preceded by a long study of this tool. Part of the time we spent searching for an analogue of the functions we were used to from CCNet. Next will be described interesting points from the work done.

Our claims for CCNet

1. CCNet no longer develops. You can still use it, but you will have to expand the functionality and correct existing / potential errors on your own.
2. The Source Code Management (SCM) polling mode is unstable about changes, namely, for automatic launch when there are changes in the version control system. If there are problems with the network, in this mode the project receives the status “Failed”, even if it did not start. In practice, the problem occurs so often (unfortunately, in our office is not the most stable Internet access), it becomes impossible to use this mode.
3. When polling SCM about changes, if there is an error in the version control system (for example, if any directory specified in the settings was removed from the repository and a tree conflict arose), the project is immediately interrupted, keeping the status “Success” - the project stops working but its status in the web interface and desktop utility remains "green". In this mode, for example, running tests may not be performed for weeks, and there is a risk that no one pays attention to this, thinking that the tests work successfully.
4. The general server log is too verbose and unstructured: it is difficult to understand which build step has fallen off and to find the log for this step. When running several projects in parallel, the build log is “mixed”. The XML log of a separate project is available in the web interface, but on the contrary, it is often not sufficiently detailed and does not contain all the commands that are run.
5. Inefficient parallelization of subtasks within the project. Subtasks are run in parallel in groups by the number of processor cores. If the group includes long-lasting and quick tasks, then new tasks will not be launched until all tasks from the previous launch have completed.

Comparison of use cases

Server settings

The CCNet project settings (server configuration) were stored in one xml file, and various passwords in another. Although the settings file has reached ~ 4500 lines, it was quite convenient to use. With an easy click of Alt + 2 in Notepad ++, the list of all projects can be minimized and edited to the desired one (Figure 1).

Figure 1 - Editing CCNet settings in Notepad ++

Although the file contained duplicate code, there was no particular difficulty with server support.

This is how the SCM block was filled:

<svn> <username>&SVN_USERNAME;</username> <password>&SVN_PASSWORD;</password> <trunkUrl>&SVN_ROOT;...</trunkUrl> <workingDirectory>&PROJECT_ROOT;...</workingDirectory> <executable>&SVN_FOLDER;</executable> <deleteObstructions>true</deleteObstructions> <cleanUp>true</cleanUp> <revert>true</revert> <timeout units="minutes">30</timeout> </svn> 

So the MSBuild block was filled:

 <msbuild> <description>PVS-Studio 2015</description> <workingDirectory>&PROJECT_ROOT;...</workingDirectory> <projectFile>...\PVS-Studio-vs2015.sln</projectFile> <buildArgs>/p:Configuration=Release</buildArgs> <targets>Build</targets> <timeout>600</timeout> <executable>&MSBUILD14_PATH;</executable> </msbuild> 

And so the block was filled for common tasks:

 <exec> <description>PVS-Studio 2015 sign</description> <executable>&PROJECT_ROOT;...\SignToolWrapper.exe</executable> <baseDirectory>&PROJECT_ROOT;...</baseDirectory> <buildArgs>"&SIGNTOOL;" ... \PVS-Studio-vs2015.dll"</buildArgs> <buildTimeoutSeconds>600</buildTimeoutSeconds> </exec> 

On the basis of such a CCNet project file, then it conveniently displays all the steps performed (albeit, only in the desktop tray utility. For some reason, the Web interface did not support this). In Jenkins, we had to tinker with the “high-level” display of the stages of the integration project, but this will be discussed later.

For Jenkins, you need to store a lot of configuration files: server config, configuration files of some plugins, each project has its own configuration file. Although all these files are in xml format, they are not very convenient for viewing and editing (at least compared to CCNet), since All commands within tags are spelled out in solid text. True, this is largely due to the ideology of using the tool. The configuration is written to CCNet manually, and therefore, it can be “beautifully” formatted. Jenkins, on the other hand, assumes editing project settings via its web interface, and automatically generates configs.

This is what the commands in Jenkins configurations look like:

 <hudson.tasks.BatchFile> <command>CD &quot;%BUILD_FOLDERS%\Builder&quot;&#xd; PVS-Studio_setup.exe /VERYSILENT /SUPPRESSMSGBOXES ...&#xd; Publisher_setup.exe /VERYSILENT /SUPPRESSMSGBOXES</command> </hudson.tasks.BatchFile> 

And this is a very small example.

View task statuses

As I wrote earlier, in CCNet projects are filled with Task blocks. This is what a successfully completed task looks like with the display of steps (Figure 2).

Figure 2 - Viewing task status in CCTray (desktop client for CCNet)

The error in any of the blocks can be clearly seen in the hierarchy of subtasks. This is a very convenient and clear visualization of the integration process. Almost never there was a need to look for logs; by the description of the task it was immediately clear what to check on the local computer. There was nothing right in this form in Jenkins, so I had to study this moment in detail before moving to a new server.

One can draw such an analogy between CCNet and Jenkins: there is a project in CCNet (Project), a step in this project is a Task (as seen in the figure above). Jenkins also has a project (Job), and his steps are Step's (Figure 3).

Figure 3 - Correspondence of project naming in CCNet and Jenkins

Unfortunately, Jenkins web interface does not know how to visualize the work of individual steps - Job has only a complete console log of all steps together. The big inconvenience here is that from Jenkins it is impossible to see which of the steps ended with an error - you need to look at the complete Job’s build log. And since you quickly get used to the good, you didn’t want to give up the old usage scenario. Then Multijob Plugin came to our aid .

This plugin allowed us to make the following innovations:

1. Using Job'ov as Step'ov in other Job'ah. Thus, universal Job'y appeared, which allowed to separate the log of specific subtasks and, most importantly, separately view the statuses of specific subtasks. Jenkins Web interface is able to visualize well the performance of individual jobs within Multijob - just what we were looking for. Figure 4 shows an example of a Multijob run.

Figure 4 - View completed Multijob

2. Using universal Job'ov managed to get rid of duplicate code. For example, there is a compilation of some utility: for distribution, to run tests and to run code analysis. In CCNet, these were the same Task blocks in 3 different projects. Jenkins compiled this utility with Job, which is used by several Multijobs.

3. When creating projects in Jenkins, the following ideology is used. All jobs are divided into multijob “projects” and universal “steps”. The names of universal jobs are prefixed with “job_” and do not imply use as an independent project. Also, they do not contain the download of source code from the repository. Multijob names are prefixed with “proj_” and include downloading the source code, and running only other jobs. (Step'y we try to avoid, because they are not visualized).

Universal jobs are launched with the following parameter:

 WORKSPACE=$WORKSPACE 

This means that Job will be launched in the working directory of the Multijob.

Thus, it is possible to obtain separately the update log of the source files and the logs of all stages of the assembly separately. To follow this ideology for all projects is difficult and meaningless. This is done only for a few of the largest and most important projects that need to be studied in detail when problems arise.

4. In Multijob, you can configure conditional and parallel runs of jobs. Multijobs can run Multijobs using the same rules. So you can combine project launches: for example, run the assembly of all installers or all tests.

View build logs

It was extremely inconvenient to view the project build logs in CCNet, since they mixed up with the server output, and had special markup. In Jenkins, there is no such problem, and in addition there is an opportunity to share logs for subtasks in some projects.

Getting a source code revision

Jenkins has its own revision version for each added link to the SVN repository. Those. If you add several directories, the numbers can be very different, and you need one maximum.

According to the documentation, it’s necessary to work with it as follows:

If you have multiple modules checked out, use the svnversion command. If you have multiple commands checked out, you can use the environment variables, where you’re configured.

And they did it: from all the SVN_REVISION_ <n> values, the maximum is taken and added to the assembled programs.

Useful Jenkins Plugins

Expansion of possibilities with the help of plug-ins allows the most flexible configuration of the server. The plug-ins for running tests in Visual Studio are perhaps the only ones that we refused to use. They had additional required launch parameters that we didn’t use, so it was easier to make a generic Job, just running tests from the command line. Unfortunately, “out of the box” Jenkins didn’t know much of what we were used to at CCNet. However, with the help of plug-ins, we managed to “return” all the required functionality.

The following is a list of plugins with a small description, which was convenient for us to use:

1. Multijob plugin - allows you to use as a build stages other jobs with the possibility of sequential and parallel execution.
2. Environment Injector Plugin - with this plugin you can set global passwords. They are used as environment variables, while the plugin hides the values ​​of such a variable in the log.
3. Pre SCM BuildStep Plugin - add extra steps before executing version control system commands.
4. MSBuild Plugin is a handy plugin for building projects using MSBuild. In the settings, the paths to different versions of MSBuild are indicated once. Next in the project, you can easily add assembly steps.
5. Parameterized Trigger plugin - adds project launch parameters. You can, for example, make a choice of the trunk / stable branch for building the distribution.
6. Post-Build Script Plug-in - perform additional steps after assembly.
7. Throttle Concurrent Builds Plug-in - this plugin allows you to adjust the number of parallel running project builds globally or within a given category. It allows Jenkins to get functionality similar to queues in CCNet — the ability to execute several projects in parallel from different categories (queues), while ensuring consistent execution of projects within one queue. For example, we have the queues Installers (distributions) and Tests (tests). We want to be able to simultaneously build some distribution kit when tests are running, but tests should not work in parallel - there will not be enough "cores" on the server.
8. Build Name Setter Plugin - allowed to set the build name in the format we need: Major.Minor.Revision.Build .
9. Dashboard View - allows you to add your own display of jobs in the browser. Since we have universal jobs that don't make sense to start manually, we created a list without them using this plugin.
10. ZenTimestamp Plugin is a handy plugin that adds timestamps in the build logs.

Desktop Client Overview

To receive notifications from CCNet, we used a Windows client, CCTray.

Here are some options for working with Jenkins now:

1. CCTray - this program can be used for a server with Jenkins. Projects will look about, as before (Figure 5).

Figure 5 - CCTray Screenshot

Description as a client for Jenkins:

• Does not develop, as well as CCNet;
• Cannot show subtasks (works only for CCNet);
• Can not run projects;
• By clicking on the name you can go to the project page;
• Adjusts the type of display projects (Icons, List, Details);
• Open source.

2. CatLight (Figure 6)

Figure 6 - CatLight Screenshot

Customer Description:

• At the moment, Beta version, the final version will be paid;
• There are still crashes during installation, operation and glitches in the interface;
• When a computer comes out of hibernation, the status of projects on the dashboard is not automatically updated;
• Cannot show subtasks for Multijobs;
• Can not run projects;
• The type of project display is not configured (the only possible view is Figure 6);
• By clicking on the name you can go to the project page;
• You can see the status of the last 5 launches and go to them;
• You can see the progress of the running project;
• When adding multiple servers, they are conveniently separated by a dash;
• There are for Windows, Linux and Mac.

3. Kato (Figure 7)

Figure 7 - Kato Screenshot

Customer Description:

• Cannot show subtasks for Multijobs;
• Able to run projects. Unfortunately, it does not support projects with parameterized launch - the utility crashes when trying to launch such a project;
• Projects from different servers are displayed in one list and are indistinguishable (not always convenient);
• Adjusts the type of display projects (List, Grid);
• By clicking on the name you can go to the project page;
• You can view the last log directly in the client, but due to the lack of monospace text this is not very convenient;
• Open source;
• Only for Windows.

4. CCMenu is an open source Mac client only. For us it is not relevant, but can someone come in handy.

Conclusion

Using CI is useful in any project. There is a great free Jenkins tool for this, which was covered in the article, as well as many other free and paid CIs. It's nice to use a developing project: for Jenkins and plugins, there are a lot of updates from time to time. New solutions are emerging, such as, for example, the new project Blue Ocean , which is still at the Beta stage, is now on the main page.

The clients for monitoring Jenkins projects have not really pleased me. There are a lot of functions that are not obvious. Perhaps desktop clients are not particularly in demand and it is more correct to use only the web interface.

When moving to a new server, it was not possible to use Jenkins as a Windows service, since UI tests are not performed in this mode. Out of position, setting up the server as a console application with a hidden window.

If you have any additions to the stated material or interesting solutions to the above problems, we will be happy if you leave a comment on the article or write us via the feedback form .


If you want to share this article with an English-speaking audience, then please use the link to the translation: Svyatoslav Razmyslov. Moving from CruiseControl.NET to Jenkins in the PVS-Studio development team