Octopus Deploy. Improving the world in a bloody enterprise

Today I want to talk about the deployment system Octopus Deploy . At the moment there is only one introductory article on this topic on Habré, so in my material I want to expand the description of the system, tell more about such important concepts as “life cycles” ( lifecycles ) and “channels” ( channels ), as well as as we have implemented and use Octopus in our work.

Who are we

We are a proprietary department of Tinkoff.ru. The team I work for is developing a cloud platform for WebOffice virtual desktops. At the moment we have a Asp.Net MVC application and a ServiceStack-based REST API on the back, and a SPA on Angular on the front. In addition, there are SOAP-services for integration with other banking systems, Jobs for performing periodic tasks and a number of microservices on .Net Core 2.0.

The problems of the old deployment process

When I came to Tinkoff, the deployment of the system was carried out using TeamCity: MSDeploy and the nth number of powershell scripts of varying complexity were used. Build configuration, responsible for the assembly of the project, and deploy were grouped by the existing circuits at the time: QA and Prod.
')
The old build and deployment process for each circuit can be conditionally presented as follows: build the project for a given environment (apply the appropriate xml configs transformations) → collect the resulting artifacts into several zip files (one for each separately deployed application + database migration scripts + deployment scripts ) → Deploy to the server specified for this environment.

Since we use several servers for fault tolerance and load distribution, the deployment process took a decent amount of time, because it went sequentially across applications and servers (writing parallel asynchronous tasks on PowerShell is rather painful). With the addition of additional contours QA2 and Preprod, the number of build-configurations grew, managing the settings of the layouts through scattered parameters was complicated.

purpose

We wanted to unify the process of deploying to different contours, speeding it up whenever possible, and also standardize the process of delivering releases depending on their type in our versioning system: major releases should follow the contours QA → Pre → Prod, minor - QA2 → Pre → Prod , hotfixes - Pre → Prod.

Octopus Deploy helped us in all of these desires. Here are the main advantages that we have gained by incorporating it into our release delivery process:

• a single deployment process for the entire application for all circuits;
• single place to store variables for the infrastructure settings of the application;
• parallel deployment to several servers;
• reducing a large variety of build configurations in TeamCity to one;
• normal (correct) build life cycle (when the same binaries are deployed first to QA, then to Preprod and then to Prod; in the version with TeamCity, a separate build configuration was configured for each environment);
• simplification of custom PowerShell scripts by discarding remoting links;
• visibility of the state of environments, deployment process, the possibility of flexible adjustment of specific deployments;
• flexible access control rights for release management.

Installation and Setup

The installation of the system itself is simple and straightforward: from external dependencies, only MS SQL Server is needed to store the database. Plus, Octopus Tentacle agent must be installed on every server you are going to deploy to.

Its installation is perfectly automated, which removes unnecessary manual work and allows you to prepare new servers for deployment. To do this, we use Ansible, and after installing the application itself, its configuration is performed by the powershell script:


As a result, after creating a new server, it is automatically registered in octopus and can be immediately deployed to it. Very comfortably.

Life cycles

One of the key entities in Octopus Deploy are lifecycles . They are used both for automatic release promotion between environments, and for restricting environments to which an application can be deployed before passing the appropriate testing.

For example, for major versions, the system life cycle is defined as QA → Pre → Prod. This means that we will not be able to finish the production version before it visits the test and preprode contours. On the other hand, the life cycle of patch versions is shorter: Pre → Prod, because they are tested immediately on the preprode.

Channels

With the help of channels ( channels ) in Octopus, you can independently deploy multiple versions of the same project. In addition to the rules limiting the package versions, for each channel you can specify the life cycle used to promote releases, configure the deployment process (because each step has a setting for which channels it should be performed), and also define the values ​​of variables. Each release created in Octopus Deploy necessarily refers to a channel.

In our project we have channels for major, minor, patch versions, as well as specialized channels for direct output to the hotfixes and deployment of an instance of the system for integration testing.

The main charm lies in the ability to automatically create releases in the right channels. To do this, we specify the range of package versions in the channel properties (and we can also use SemVer tags), which should be used to create a release in this channel. Octopus will automatically select the latest available packages.

Suppose that versions V10.1 and V11.0 are currently being tested. After the project is built from the V10.1 branch, TeamCity will upload the collected packages (the package version will be 10.1.0.xxxx) to Octopus and trigger the creation of a release in the channel of “minor” releases. At the same time, it is exactly packages of version 10.1.0.xxxx that will get into the release, despite the fact that Octopus already has newer packages with version 11.0.0.yyyy, since in Version Range, the channel of the “minor” releases has a range [10.1.0.00000, 10.1.0.99999).



Thus, different versions of the application coexist perfectly with each other in channels and follow their own deployment process to different environments.
Here is an example of how it looks in the project control panel:

Important! If you add new steps to the deployment process, add new packages to the channel rules. If no channel limit is specified for any package in the channel, Octopus will use the latest version of the package. It may happen that you applet version 10.1, but one package will be, for example, version 11.0, if it already exists in Octopus.

Creating NuGet Packages

You can create packages for deployment while assembling projects using the Octopack tool. It is installed into the necessary projects as a NuGet package, and if you are satisfied with the default settings, you do not need to do anything else.

Based on the type of project (console application, web application, windows service), Octopack itself selects the files that will be required for the deployment. If you need more fine-tuning, which files should get into the package, you can create a custom .nuspec file.

TeamCity integration

But the most interesting thing begins when automating the processes of creating (and deploying) releases after building a project on TeamCity. Here is how this process looks at us:

• the developer after reviewing his task in the version branch;
• TeamCity builds the project, determines the channel to which the assembly should fall, and causes the release to be created on the desired channel in Octopus Deploy;
• the availability tester presses the “Deploy” button in the Octopus control panel and deploys the assembly in the test environment corresponding to this channel.

Part of the task of preparing NuGet packages, pouring them into Octopus and creating a release is very easy to do by installing the Octopus Deploy plugin for TeamCity. For some other operations (for example, automatic updating Version Range at the channel) we wrote small powershell scripts.

Guided Failures

If during the deployment process an error occurs on one of the steps, the entire deployment stops and is marked as falling. This behavior is not always relevant: sometimes we are willing to come to terms with the problems in some of the deployment steps. For these cases, there is a mode called Guided Failures.

If it is on, and an error occurs during deployment, Octopus suspends all operations and displays a special window. It is possible to indicate the results of the investigation and decide whether to continue the deployment process or not. In the history of deployment there will be a note about who made it, when and what decision.

Deploy scripts in packages

Sometimes before deploying applications and services, we want to perform additional actions. For example, make sure that the application can access the port when running on behalf of the service account. This can be done using additional installation scripts, which can be written both in the Octopus web interface and also be part of the application package.

In scripts, you can use octopus variables that will be forwarded according to the variable naming rules in PowerShell (points and other special characters will be removed from variable names, thus the octopus variable MyApp.ConnectionString will be available in the script as $ MyAppConnectionString).


An example of a pre-deploy script that removes and adds a port reservation for one of our services.

Retention policy

One day, in order not to be in a situation with a fully filled hard drive, Octopus has a mechanism for removing packages that were used in older releases and are no longer needed.

A packet retention policy is described at the life cycle level, but it can be redefined for its individual phases. For example, for all application deployment phases on test environments, we store only one latest version of the package.



For the combat environment, we override this setting and store packages for the last five releases (in case you suddenly have to roll back). You can also specify for which releases to store packages on the machines themselves. In the case of a rollback, you can deploy it faster without spending time re-downloading packages from the Octopus server.

Additional nishtyaki

In this section I want to share some powershell scripts that we use in our system. You can use them as is or as an example to implement your own ideas.

Conclusion

We are very pleased that a year ago we started using Octopus Deploy, and after a short period of test operation, we completely switched to it to deploy the system on all of our environments.

The project is actively developing, minor releases are released almost every week. In the latest versions of Octopus Deploy, I learned how to deploy .net core and mono applications to linux machines, deploy docker containers, and in the latest release 3.17 they added first-class support for deploying Java applications to Tomcat, RedHat JBOSS and Wildfly application servers.

In the next article I will talk about multi-tenant deployments, which we use to organize the deployment process of the feature brunches to individual virtuals, automatically raised in the local OpenStack cloud, to conduct isolated testing of individual tasks.