Personal experience: how our system Continuous Integration looks

We at Positive Technologies not only conduct security research on various IT systems, but also develop products that help detect and prevent threats, as well as minimize damage from possible attacks.

Over the past few years, our product line has seriously expanded - to the many well-known MaxPatrol systems on the market, a number of new tools have been added, from application-level firewalls to incident management tools. This development has put us before the need to adapt the development processes in the company - so we are actively introducing DevOps practices and related technologies into our work.

Today we want to tell you about the model of the Continuous Integration system we created.

Prehistory

Many years ago, we chose TFS as the Continuous Integration system for automating the assembly and testing of code. Over time, it became clear to us that this system has a number of shortcomings. In particular, when using it:
')

• It is difficult to maintain patterns of assembly, deployment and test configurations.
• There are problems with the integration of non-C # projects.
• Impossible operational expansion of infrastructure.

The longer we used it, the greater the need for typing and templating the creation of all types of configurations, speeding up the creation of standard projects in our Continuous Integration systems, and ensuring project extensibility while simplifying the addition of new configurations.

It took us almost two years to solve these problems. This is what the Continuous Integration infrastructure of Positive Technologies looks like now. It consists of a bundle of three basic services:

• TeamCity is the main organization of Continuous Integration.
• GitLab - source code storage system.
• Artifactory is a storage system for collected binary versions of components and products.

We paid special attention to the development of model projects for a system of continuous integration. This allowed us to achieve unification of projects, highlighting the so-called release scheme of assemblies with promotions in TeamCity.

Here is how it works. All projects look the same: they include the configuration of the assemblies that fall into the artifactories, after which they are deployed, tested and promoted to the release repository of the project.



As a result, all projects have a standard three-level organization. The first level is the project level, for example, TeamCity stores various assembly templates at this level, followed by a subproject level that includes various components of a common product, and each subproject includes standard configuration groups for assembling, deployment, testing and tools.

As a result, now all the projects in our TeamCity have the same hierarchy, which is very convenient. Read more about it here .

What is the result

We have been developing the Continuous Integration system for almost two years now and now it looks like this. In addition to the standard configuration groups for assembling, deploying, testing and promoting assemblies, we now have a system for publishing tested release assemblies on the Global Update server, from where they extend further down to the customer’s infrastructure.



High-level IDEF0-model of Continuous Integration processes in Positive Technologies for 2016. By clicking the picture will open in full size.

In addition, we use a number of other technologies, including Docker, SaltStack, TeamCity, Teampass, TestRail, VMware, Zabbix and others.

However, despite all the advantages of unification, the system we created at the first stage had its drawbacks.

Not so simple

First of all, the configuration logic in TeamCity was quite complex, which made it difficult to work. These configurations were supported only by the company's DevOps team, and very soon we reached the limits of project scaling when working in this format. However, this problem was partially solved by creating scripts for automatic generation of sample projects.

We also lacked delivery and installation mechanisms for products integrated with our Continuous Integration system. The inconvenience was also caused by the fact that the assembly processes themselves on the assembly servers and the developers' machines were different - and we could not afford to ensure their “sameness”.

It became clear that we need to move on and develop our system.

Plans

We are planning to create two build pools of machines for Windows and Linux based on TeamCity. Further development of an assembly process optimization system called CrossBuilder is also expected. With it, you can solve a number of tasks:

• Provide identical build processes on build servers and development machines.
• The ability to use different CI-systems.
• The declarative description of the build process using special manifest files is delegated to development teams, freeing up the time of DevOps employees.

Solving these problems will allow us to further improve the efficiency of our Continuous Integration system.

That's all for today, thank you for your attention! In the comments, we will be happy to hear comments on the solutions we have chosen, share your experience in building Continuous Integration systems!

PS The story about our system of Continuous Integration was presented in the framework of DevOps-mitap, which took place recently in Moscow.

Video:



Slides



The link presents presentations of 16 reports presented during the event. All presentations and video presentations will be added to the table at the end of this topic-announcement .

Author: Timur Gilmullin