Continuous Delivery: Continued

2 years ago in the article “Continuous Delivery & Sitecore: Our Implementation” I introduced our system for the automatic delivery of Web applications based on Sitecore. These were the first small steps towards the world of full automation.

Since then, the project has not only not died, but has grown and matured. Moreover, it is still developing and incorporating more and more tools and opportunities. Perhaps the only thing that remained unchanged is the version control system (GIT) and the branch system.

Major changes that have befallen the project (I will dwell on each in detail):

• Delivery via SSH is basic only for projects living on Linux and Windows 2003. Projects hosted on Windows 2008 R2 and higher use Web deploy.
• Since, in addition to web applications on Sitecore CMS, we began to develop on Umbraco, Episerver and just ASP.NET web applications, services and applications - now the CD also works with them
• The internal Nuget gallery and the update of the open source libraries we are developing on Nuget.org are also delivered using TeamCity

SSH delivery for Linux php projects

In fact, everything is easier than ever. Since PHP does not require compilation, the delivery is to pick up the sources from Gita, and, having picked up via SSH to the target server, upload them.
In TeamCity for this there is a special plugin - deploy-runner, which, strictly speaking, deals with the entire delivery. In the future, we are planning to attach integration tests and unit tests to this type of builds. Currently used only for internal testing.
')

Nuget, SymbolSource, packaging and updating

Inside our network, we expanded the Nuget gallery (fork from https://github.com/NuGet/NuGetGallery ), the SymbolSource server for this gallery ( http://www.xavierdecoster.com/setting-up-your-own-symbolsource-server -step-by-step ). Since this is also a web application - Teamcity is responsible not only for their content, but also for their updates (monitors our repositories forks, and when updating the fork - delivers updates to the web application using the steps described below).
TeamCity is also responsible for publishing Nuget packages to the gallery (both internal and external), updating packages using the tools built in TeamCity for packaging and publishing packages. Also, each package is published in the corresponding SymbolSource gallery (internal or Symbolsource.org).

MsBuild, MsDeploy and everything that revolves around ...

In fact, the delivery itself and everything connected with it.
I would like to say a few words about MsDeploy and the reason for leaving SSH. Despite the fact that delivery via SSH is quite fast, preparatory operations require a lot of time and expensive disk operations. Plus, we always in any build should either deal with the differentiation of files for delivery (in order to reduce the size of the package being sent), or always deliver the entire application (which does not quite positively affect the published package). If we take MsDeploy, which is called by Visual Studio or MsBuild, then MsDeploy is responsible for delivering files to a web application, it also evaluates whether it is necessary to update / delete / leave a specific file, which positively affects the speed of delivery. Also, in the default configuration, MsDeploy takes over and establishing order - if the file was deleted / deleted in the IDE, then it is not needed in the target web application.
To ensure stable and manageable delivery work, the whole process revolves around the MsBuild engine, whose behavior is controlled by a set of imported projects (targets files), unified by the procedures performed and differing only by project type (Sitecore, Umbraco, Episerver or web application without CMS).
I’ll focus on target files in more detail ...

Sitecore

The Sitecore.targets file describes the following steps:

1. Preparing an application for delivering logical structures to the database - publishing a configuration file that allows the use of any symbols and disables the most resource-intensive operations that are carried out by Sitecore when publishing. This step requires restarting the application and is therefore called only if we have something to deliver to the database.
2. Revitalizing the application with Wget after restart (in fact, just a request for the main page of the site)
3. Adding the latest version build that implements CI to the application
4. Static transformation of the web.config file, for garbage collection, which remains in the temp folder after delivery, by means of Sitecore itself
5. Import the sharedtargets file, which is responsible for all other operations during the build, with the help of its target. And the order of accomplishment of goals in the sharedtargets file is controlled by the AfterTargets and BeforeTatgets directives.
6. Which folders to delete during the build of the application (the list can be expanded in the csproj file of the application itself) is necessary because we do not use self-cleaning files (SkipExtraFilesOnServer = true), in order not to clear the content files potentially created by the application

Episerver

The Episerver.targets file describes the following steps and delivery requirements:

1. Add an assembly of the latest version that implements CI in the application
2. Import the sharedtargets file, which is responsible for all other operations during the build, with the help of its target. And the order of accomplishment of goals in the sharedtargets file is controlled by the AfterTargets and BeforeTatgets directives.
3. Which folders to delete during the build of the application

Umbraco

The Umbraco.targets file describes the following steps and delivery requirements:

1. Add an assembly of the latest version that implements CI in the application
2. Which folders to delete during the build of the application
3. Import the sharedtargets file, which is responsible for all other operations during the build, with the help of its target. And the order of accomplishment of goals in the sharedtargets file is controlled by the AfterTargets and BeforeTatgets directives.
4. Updating the application version in ClientDependency.config (Umbraco cleverly caches static resources)

Build itself

All builds are unified and can differ only according to the requirements of the CMS or the consumer. Here I will try to make out all the steps in order (I will indicate in brackets at the beginning of the description of the step if this step is specific to some target)

1. Informational e-mail sent to the group that the build is running. Contains the following information:
   - for which project and which environment is the build started
   - who started the build
   - what changes are included in the build (comment to the Chekines in the repository, if the comment starts with #number, then #number is framed by a link to the request in our tracking system)
2. (Sitecore) Check to see if there are any pekedges in the build (logical elements of the Sitecore database for delivery). If there is, update the application (put a special configuration file, update the CI assembly) and animate it.
3. (Sitecore) Delivery pekedgey using WCF service
4. (Sitecore) Publish using WCF service
5. (Microsoft Network Load Balancer) If the application is hosted on several servers and the traffic is balanced by Microsoft NLB - then in this step we turn off the application from balancing with the help of Drainstop (the new traffic to the application does not fall. We end up servicing those users that were at the moment start drainstop)
6. Delivery of App_offline.htm to the root of the application (so that users are not afraid of errors) and disabling cloud monitoring services (so that Uptime does not fall, because the ASP.NET application with App_offline.htm is at the root, although it shows a beautiful picture and an inscription, but returns HTTP status 503). What monitoring services to disable is controlled by the build parameters
7. Nuget restore - despite the fact that you can and should enable Nuget restore at the solution level, some assemblies change the build process itself (for example, Microsoft.Bcl.Build)
8. MsBuild Publish - here all the magic happens ...
   I will not describe everything that MsBuild does, but only describe the changes that are made by the sharedtargets.target file.
   1) If the application-specific target file adds CI files - they are added at the very beginning (we virtually mark them as Content with meta data CopyToOutputDirectory = Always)
   2) AddVersionTxt - adds the version.txt file to the root of the web application, which contains the build number and date (convenient for testers and third parties - what is now published). In Sitecore, it is also visible in the application information.
   3) (Sitecore) Static transform for deleting files from the temp folder
   4) Encrypt - if the RSA key is added to the configurable folder in the application, and the EncryptWebConfigOnTeamcityBuild flag is true, then during the build TeamCity will encrypt the appSettings and connectionStrings sections in the web.config (you can add more sections at the csproj level of the application file)
   5) If an additional database is used in the application, then SqlUpdaterSupportTarget is called. In the same way as the second step, it checks if there were any changes in the configured folder and if there were any - by means of MsDeploy it updates the database configured in TeamCity. Able to work with both MSSQL and MySQL. So far, only one database can be updated. If there are several, you need to add an additional step before deleting App_offline
9. (EpiServer) Since Episerver adds the files necessary for the CMS outside the root of the web application to the AppData folder and goes to them via VirualPathProvider - we have to check if there have been any changes in this folder, and if so, upload them to the server
10. Uninstalling App_offline, revitalizing the application and turning on monitoring services back
11. Web application revitalization - Wget runs through all the first-level pages of sites specified in the build configuration.
    In the near future, instead of the build configuration, this task is planned to be put into the WCF service.
    Also, at this step, the numerical values ​​are saved (how long it took for it). In the future, we will conduct a very rough analysis of the performance of a web application (lamer, but metric :))
12. Analysis of the published web.config using OWASP (report is available in Teamcity reports)
13. Standard web application checks and sending an email with the results of the checks and the build to the group.
    At the moment, standard checks include only checking for robots.txt, analyzing it using the https://code.google.com/p/robotstxt/ parser, and sitemap.xml validation.
    This step should and will be expanded by various kinds of integration and unit tests.

In addition to web applications, we also develop additional services for Windows. And they also need to be delivered ...
There is also a separate build for this, consisting of the following steps:

1. Nuget Restore
2. Build with Visual Studio (Visual Studio (sln) runner)
3. Stopping and deleting services on the target server using the MsDeploy runCommand provider
4. Filling files with MsDeploy
5. Installing services on the target server using MsDeploy runCommand

Well, another build is needed for TeamCity to build for itself all the tools described above. They are all assembled into one solution, and by changing it, a build is started consisting of the following steps:

1. Nuget Restore
2. Visual studio build
3. Powershell script that decomposes the results of step two for all agents

In addition to these builds delivering the code, there is another TeamCity build - not related to the delivery of the code, but simply a service one. It consists of two steps and its only purpose is to trigger the recycle of the process serving the web application. We need it in order to be able to quickly restart the web application.

1. MsBuild, using the recycleApp provider built into MsDeploy, causes the application to restart
2. Wget in spider mode loads the first page of the web application (for its speedy recovery)

This "build" is not often called, as this is an extreme measure.

Visualization

As a final touch, with the help of open-source Tula Gource , I added CI development visualization: