Geekly Articles each Day
📜 ⬆️ ⬇️

Automation of development processes: how we in Positive Technologies implemented DevOps ideas



Hi, Habr! In this article, I’ll tell you how we at Positive Technologies created and developed the development automation department, implemented DevOps ideas into development practice and what tools and technologies were used for this, as well as how CI / CD processes were organized. Including share successes and plans for the future.

Who we are and what we do


Positive Technologies is a multi-product vendor in the field of information security. Positive Technologies products are delivered according to traditional models (in the form of installed software), SaaS, virtual machines and “bare metal appliance”. Our department, informally called DevOps, is responsible for automating and supporting the development processes of all our products: the XSpider network scanner, MaxPatrol 8 security and compliance standards, the MaxPatrol SIEM security and event management solutions, and cyber security incident management systems APCS - PT the ISIM , the shield level applications PT the application Firewall , a code analyzer PT the application the Inspector , a multithreaded system for identifying malicious content PT MultiScanner and all other Company products .
')
We believe that the work on the principles of DevOps and the responsibilities of the relevant automation departments are highly dependent on the specifics of the specific company. The specifics of our company imposes on us the obligation to organize the delivery of a boxed product to end customers and support several versions of each product at the same time.

At the moment, our department is responsible for the entire conveyor for mass production of the company's products and related infrastructure, ranging from assembling individual components of products and integration assemblies to sending them to testing on our servers and delivering updates to the geographically distributed infrastructure of update servers and then to the infrastructure customers. Despite the large volume of tasks, only 16 people cope with them (10 people solve Continuous Integration tasks and 6 - Continuous Delivery tasks), while the number of people involved in product development is more than 400.

The key principle for us is “DevOps as a Service”. DevOps in modern IT has long stood out in an independent engineering discipline - very dynamic and technological. And we must quickly bring these technologies to our developers and testers. In addition, we make their work more comfortable, efficient and productive. We also develop internal support tools, regulate and automate development processes, preserve and distribute technological knowledge within the company, conduct technical workshops and do much more useful things.

About our first steps in the development of DevOps ideas in our company can be found in the article “ Mission is feasible: how to develop DevOps in a company with many projects ”.

Continuous Integration and Continuous Delivery Model


When we at Positive Technologies decided to develop the ideas of DevOps, we were faced with the need to embed them in situations in which dozens of teams simultaneously worked on projects both public and non-public. They were very different in size, using different release models and technological stack. We understand that centralized solutions and a high level of organization of development automation are better than the “creative chaos” of ordinary development. In addition, any automation is designed to reduce the cost of product development and implementation. In companies where there is a dedicated service for implementing DevOps techniques, the serialization of all implemented tools is ensured, so that developers can work more efficiently using ready-made automation templates.

Continuous Integration is just one of the processes that connect product developers and end users. The infrastructure of Continuous Integration in Positive Technologies has developed around a bundle of three basic services:


Historically, we chose GitLab, TeamCity, Artifactory to store code, organize assemblies and store their artifacts. We started using GitLab a long time ago because of its convenience compared to svn, which we had before. The advantages of GitLab were that it made it easier to apply changes (merge). As for the Artifactory, there is no serious alternative for storing binaries (components and installers). However, you can use file balls or MS SharePoint, but this is more likely for those who are not looking for easy ways, and not for automatists. We switched to TeamCity after a long period of operation and builds on MS TFS. The move was tested for a long time, run-in on small products, and only then it was spent completely. The main reason is the need for simple typical solutions and scalability of the assembly and test infrastructure for multi-component products written in various programming languages. TFS couldn't give us that.

We paid special attention to the development of model projects for a system of continuous integration. This allowed us to achieve the unification of projects, highlighting the so-called release scheme of assemblies with promotions in TeamCity. At the same time, all projects look the same: they include configurations for assemblies whose artifacts fall into the Artifactory, from there you can take them for deployment, testing and promotion to the release repository of the project (see the diagram in Figure 1). We wrote more about the terminology used and the organization of the standard CI process in the article “ DevOps best practices: recommendations on the organization of configurations in TeamCity ”.



Figure 1. Basic elements of a typical project for a release scheme of assemblies with promotions (one click will open in full size)

We have developed our Continuous Integration system for more than two years, and now, in addition to standard configurations for assembling, deploying, testing and promoting assemblies, we have added it with the Continuous Delivery system developed by us, called SupplyLab, to publish the tested release assemblies to Global Update servers where they come from and propagate through Front Update-servers further, up to the infrastructure of customers, where they are deployed and updated.

As noted above, all solutions provided by our DevOps-team are typical, scalable and built on templates. Although the infrastructure requirements, programming languages ​​and build algorithms used by the teams differ, the overall concept of the CI / CD process remains unchanged (see the diagram in Figure 2):

commit in git - automatic build - deploy to test servers - functional and other types of autotesting - promoting assembly to stable - publishing to GUS - distributing through FUS on the customer's infrastructure - installing or updating the product on specific servers.

At each of these stages, the DevOps department helps developers solve specific tasks:


We monitor each stage through a monitoring system. In case of failure, developers can write a request to our service, we will try to localize the problem and fix it.



Figure 2. High-level IDEF0-model of Continuous Integration and Continuous Delivery processes in Positive Technologies

Read more about how we built and developed our CI / CD system at the beginning of 2017, in the article “ Personal experience: what our Continuous Integration system looks like ”.

A little bit about monitoring


In our team, those responsible for the direction of work related to monitoring are allocated: from the application for connecting a server or service to monitoring, to the final beautiful dashboard management systems (charts, metrics, warning systems) provided to the teams. The functional responsibilities of this group are: setting up and providing a monitoring service as a service, providing standard templates for monitoring servers and services of various types, designing a hierarchical monitoring system, predicting a lack of resources.

We use the Zabbix monitoring system. To simplify the monitoring services, we decided to separate the areas of responsibility between the DevOps team, developers and testers, and also wrote our own tools for interacting with Zabbix ( zabbixtools ) and using the “monitoring as code” approach. Let us clarify with an example: so, if we have a test bench that needs to be monitored, then the tester assigns him a certain role. Each host — only one role in which there can be several profiles for monitoring processes, services, APIs and others — is provided by the DevOps team (see the relationship between the Zabbix entities in Figure 3).



Figure 3. Model of relationships between entities in Zabbix

To simplify the configuration, a system is implemented in which Zabbix collects as much data about the observed indicators from the target server as possible. To do this, zabbixtools uses Low Level Discovery (LLD) functionality. This allows you to make any monitoring settings on the monitored server itself, and not in the Zabbix admin panel. Configuration files are saved and updated via git.

You can read about the implementation of the monitoring system in the article “ Zabbix for DevOps: how we implemented the monitoring system in the development and testing processes ”.

Deficiencies in the functionality of the CI / CD solutions we use


We identify all the problems and shortcomings of the elements of our CI / CD systems in our knowledge base DevOps. It is quite difficult to summarize the problems of each specific service within one article, therefore I will only mention the most indicative, for example, for the CI system TeamCity.

At the moment, all assembly configurations in TeamCity are accompanied only by our department. We make changes to them at the request of the developers to avoid problems with the breakdown of their configurations due to ignorance of our patterns and tools. Also, our team monitors the environment on the assembly servers - agents of TeamCity. Developers do not have access to them. We have limited it in order to avoid making unauthorized changes on the servers and to simplify the localization of possible problems and debug builds: once there is a guarantee that the configuration and environment do not change without authorization, then the problem may be either in the code or the network. It turns out that we cannot provide developers with the opportunity to independently debug the assembly process on TeamCity and tune the environment on the assembly agent.

Thus, the first problem is that we lack the ability to delegate the build process to teams using the “build as a code” approach. TeamCity does not have an easy-to-use DSL (specific assembly description language). I would like the assembly description code to be stored in the repository next to the product code, as, for example, in Travis CI. Now we solve this problem by developing our build system - CrossBuilder. It is planned that it will provide developers with the opportunity to give a simple description of the assembly, store and modify it in their project, and will also not depend on the CI system used, implementing the assembly through a system of plug-ins. In addition, this system will allow to run assemblies locally, via a standalone client.

The second problem with making changes to the build environment, like many other engineers, we solved for Linux with the transition to docker-images, for the preparation of which, including the storage of dockerfiles, the teams are responsible for themselves. This allowed us to allocate a pool of similar Linux build servers, each of which can start building any command in its own environment. For Windows, we are just starting to work on a similar scheme and are now conducting experiments.

Our toolkit


We have tried many different services and tools, and over time an evolutionary path has developed their specific set. To solve the problems of our department, we use: TeamCity and GitLab CI as CI systems, GitLab for storing code, Artifactory as binary artifact storage and proxying repository, SaltStack for automating deployment scenarios, Docker for isolated build quality, SonarQube for analyzing code quality, UpSource for conducting a command code review, TeamPass for securely storing secrets, VMware as a virtualization tool, Zabbix for monitoring our entire infrastructure, TestRail for storing test results, Kubernetes for managing Linux containers some others.

From the corporate standard email clients, we use Outlook / OWA and Skype for Business, for the rest we are not limited by the rules, so everyone uses what they find convenient: Chrome, Thunderbird (Mozilla), Opera browsers; file managers Total Commander, FAR, ConEmu and the usual shell; editors MS Word, Notepad ++ and, of course, vim; ssh clients Putty, WinSCP; sniffers and traffic analyzers (tcpdump, windump, wireshark, burpsuite etc); from IDE like PyCharm, Visual Studio, WebStorm; For virtualization, we use vSphere Client and VirtualBox.

Following the developers of our company, we have to automate processes mainly under Linux (Debian, Ubuntu) and Windows (2003-2016). Accordingly, the language expertise we have is the same as that of the developers - this is Python, C #, batch / bash. For the development of standard modules, for example, scripts-metaranners for TeamCity, we have a regulation that clearly describes all development steps: starting from naming scripts, methods, code-style and ending with the rules for preparing unit tests for a new module and functional testing of the entire assembly devops-tools (our internal automation scripts). In the process of developing scripts, we follow the standard git-flow model with release and feature builds. Before merging branches, the code in them necessarily passes the code review: automatic, with the help of SonarQube, and manual from colleagues.

Open Community Open DevOps Community on GitHub


We do not see any particular advantages for organizing CI / CD processes based on open-source solutions. There are a lot of controversies on this topic, but each time you have to choose which is better: a box solution, tailored to specific goals, or a “as is” solution, which will have to be finished and maintained, but providing wide possibilities for customization. However, last year at the end of our mitap Op! DevOps! In 2016, we talked about the fact that we were allowed to upload a part of the company's code of tools in open-source. Then we just announced the community DevOps-developers Open DevOps Community . And already this year, at the Op! DevOps! 2017 we have summed up the interim results on its development.

In this community we are trying to unite the work of various specialists into a single system of best practices, knowledge, tools and documentation. We want to share with colleagues what we can do ourselves, and also to learn from their experience. Agree, because it is so useful to discuss a difficult task with an understanding colleague. Everything we do is open tools. We invite everyone to use them, improve, share knowledge and approaches in DevOps. If you have ideas or tools to automate anything, let's share them through the MIT-licensed Open DevOps Community! It is fashionable, honorable, prestigious.

The goal of the Open DevOps Community is to create open, off-the-shelf solutions for managing the full cycle of the development, testing and related processes, as well as the delivery, deployment and licensing of complex, multi-component products.

At the moment, the community is in the initial stage of its development, but now you can find some useful tools written in Python in it. Yes, we love him.


Each tool has an automatic build in Travis CI with a display in a PyPI repository, where they can be found and installed via a standard mechanism: python pip install.

Several more tools are being prepared for publication:


We invite everyone to participate in the development of the community! We have a sample project ExampleProject , which contains the general structure and detailed instructions for creating your own project in the community. It is enough to copy it and make your project by analogy (see the steps for preparation in Figure 4).



Figure 4. A few simple steps to prepare your tool for publication in the Open DevOps Community

Retrospective and Future Plans


The year 2017 is coming to an end and it is already possible to conduct a small retrospective analysis of the work done by our team to develop DevOps ideas at Positive Technologies:

  1. 2014 - the realization that the technologies we have then (svn, SharePoint, TFS) cannot go far, after which work began on researching and piloting new CI / CD systems.
  2. 2015 - standard scenarios and processes were prepared and configured, the DevOps system framework was built on the basic teamCity + GitLab + Artifactory link (for more details, see the video ).
  3. 2016 - an active increase in the volumes of assembly and test configurations (up to +200 per month!), Transfer of all processes to a standard release scheme of assemblies with promotions, ensuring stability and resiliency of the assembly infrastructure.
  4. 2017 - consolidation of success and stabilization of the growth of projects, high-quality transition to the usability of all services provided by the DevOps team. A little more detail:
    • According to statistics, only in November 2017 for ~ 4800 active assemblies currently available on our infrastructure ~ 110,000 assemblies were launched, with a total duration of ~ 38 months, an average of 6.5 minutes each. on the assembly;
    • In 2017, for the first time in our department, we started with an annual plan, which included non-standard tasks that go beyond the usual routine that a single engineer can cope with, and the solution of which required considerable labor costs, joint efforts and expertise;
    • completed the translation of the build environments to docker and allocated two unified pools of builders for windows and linux, came close to the description of assemblies and infrastructure on DSL (for paradigms build as a code and infrastructure as code);
    • stabilized the delivery system for updates of SupplyLab (some statistics on it: customers pumped out ~ 80TB of updates, 20 product releases and ~ 2000 service packs were published).
    • This year we began to consider our production processes from the point of view of the technological chains used and the so-called final useful result (CRC).

So why do we need an automation department in our company? We, like other divisions, work for the final useful result, therefore the main goal pursued from the implementation of DevOps ideas in our production processes is to ensure a consistent reduction in the cost of production of the CRC.

As the main function of our DevOps department, we see the macro-assembly of individual parts into a single useful product and the reduction of the cost of the chain: production - delivery - software deployment.

We have a grocery company and, based on its future needs, global tasks have been defined for our department for 2018:

  1. Ensuring the stability of development processes by:
    • compliance with assembly SLA and time to solve typical tasks
    • profiling and optimization of “bottlenecks” in all areas of work and processes in teams;
    • accelerate the localization of problems in complex production chains due to more accurate diagnosis and monitoring).
  2. Regular webinars on existing practices to reuse solutions in products and ensure serial production.
  3. Transfer to serial duplication of processes and tools in teams. For example, speeding up the preparation of standard assembly projects, primarily due to CrossBuilder.
  4. Commissioning of the management system for the composition of the release and the quality of the assembled components and installers, due to the use of the CrossPM and DevOpsLab features.
  5. DevOpsLab development - automation system and delegation of typical tasks to project teams. For example, promotion of packages and installers, generation of typical assembly, deployment and test projects, resource management of projects, issuance of rights, setting quality labels for components and installers.
  6. Development of a standard and replicable process for the delivery of our products through the SupplyLab update system.
  7. Transfer of the entire infrastructure to the use of the paradigm infrastructure as code. For example, prepare standard scenarios for deploying virtual machines and their environments, develop a classification of machines according to consumed resources, accounting and optimization of the use of virtual resources.

Instead of conclusion


We are sometimes asked what we especially remember in our work, which situations from our practice of implementing DevOps were remarkable and what. However, it is difficult to single out any one case, because, as a rule, abrarals and their resolution are remembered. We do not welcome avraly, but we like well-planned work on the built-in process - when everything is stable, predictable, expected, and in case of force majeurs, backup options are provided. According to this principle, we are trying to build our service year after year.

We are interested in working with any automation tasks: building a CI / CD, ensuring the fault tolerance of release assembly processes, monitoring and optimizing resources, and also helping people build processes in teams and developing useful tools. For us, the systematic, consistent process of building DevOps as a service is important - after all, this is striving for the best.

PS And here are some of our Telegram stickers for DevOps .

The article is based on an interview in the journal System Administrator and a report on Op! DevOps! 2017 ( slides ).

Posted by Timur Gilmullin , Development Process Support Team Lead, Positive Technologies

Source: https://habr.com/ru/post/343884/

More articles:


All Articles