Self-configuring applications

Hello.
The introduction of the continuous integration technique confidently walks through our long-suffering homeland and more and more people are imbued with its ideas and concepts, which is very good. In this article I would like to tell you about the technique that I use at one of the stages of continuous integration - application configuration.


Photo taken with Yaplakal

Problem

Let's start with a description of the problem. We all use several environments in our work. Typically, this is the development environment (in the article we will call it Dev), the testing environment (Stage) and the production environment (Prod). They are over, maybe more. For example, local development and tester machines are also environments if your applications are deployed to them. The issue of configuring the environment is a separate big topic that we will not discuss in the framework of this article, but consider the issue of configuring the application itself. What is the difficulty? The fact that all your applications communicate with external resources, such as databases, FTP, other applications and services, and for each environment a set of such resources is unique. That is, if we conditionally take and completely copy the application from one environment to another, it will not work correctly or will not work at all. In order to run it, we need to manually edit the configuration files, and this contradicts the principles of continuous integration, which tells us that there should be no manual work, and that all processes should be automated. Automate this work in various ways. For example, store configuration files from various stands in the repository and substitute them during the deployment. There are a few problems here. First, you need to decide how to store configs from the sale, you can use a separate closed repository, but then how will the developers (or anyone else at all) make changes there? Secondly, as a rule, not only parameters of access to external resources are stored in configs, but also other parameters that do not change from stand to stand. In addition, you probably have not one application, but many, and that if you need to change the connection string to the database on some stand, you will have to edit a lot of files and try to remember which ones. As a result, the support of all configuration files in the current state becomes very time consuming.

Of course, there are various applications that are not paid and do not allow solving the configuration problem. You are free to choose any way convenient for you. But I would like to talk about the most elegant, in my opinion, and simple way to configure the application. In terms of programming languages, I use PowerShell and Python because I use .NET applications in my zoo, but I think it’s easy to create such a system for your stack. Here the idea is more important, not the technology.
')

Source of inspiration

Of course, the idea is not new and not mine, it is beautifully described in the fundamental work on continuous integration . Here are a few principles for managing application configurations from there:

• Analyze at which point in the application's life cycle it is better to enter a portion of configuration information into it — at the time of assembly, when the release candidate is packaged, during deployment or installation, at the time of launch, or at run time. Talk to the administrators and technical support team to find out their needs.
• Keep the available application configuration settings in the same location as the source code, but store the values ​​in a different location. The life cycles of configuration parameters and code are completely different, and passwords and other secret information should not be recorded at all in the version control system.
• Configuration must always be an automatic process, using values ​​retrieved from the configuration repository, so that at any time you can identify the configuration of each application in any environment.
• The configuration system should provide the application (as well as packaging, installation, and deployment scenarios) with different values ​​depending on the version of the application and the environment in which it is deployed. Each person should be able to easily see which configuration parameters are available for this version of the application in all deployment environments.
• Apply naming conventions for configuration parameters. Avoid confusing, non-informative names. Imagine a person reading a configuration file without documentation. Looking at the name of the configuration property, it must understand what it is intended for.
• Encapsulate configuration information and create a modular structure for it so that changes in one place do not affect other parts of the configuration.
• Do not repeat. Define configuration elements in such a way that each concept is presented in the set of configuration properties only once.
• Be a minimalist. Configuration information should be as simple as possible and focused on the essence of the problem being solved. Do not create unnecessary configuration properties.
• Do not complicate the configuration system. Like the configuration information, it should be as simple as possible.
• Create configuration tests performed during deployment or installation. Check the availability of services on which the application depends. Use smoke tests to ensure that each function of the application, depending on the configuration parameters, correctly perceives them.

I note that in my decision I do not take into account the version of the application, at least explicitly, in practice I did not need it, although adding this possibility is not difficult. I also do not consider the issue of testing configurations, as this is a separate topic.

Implementation

So, the main steps:

• We assemble the application on the integration server, run the tests, we get a working application configured as “default”, for example, to the Dev environment;
• We deliver the files of this application to the target server (or server);
• Run the application configuration process;
• Run the application.

It is important to note here that the same set of files must be delivered to all stands, i.e. Initially, all applications are the same, and they are already configured directly on the target server.

Now more about the configuration process:

1. The continuous integration server after delivering the application to the target booth calls the PowerShell script Configurator.ps1, which is supplied with the application. As parameters, the path to the Webconfig.json change file and the path to the stand parameters source are passed to it (more detailed below).
2. The change file is located in the same directory as the configuration file for the Web.config application (and in the same repository). File content is the description of those tags and their parameters in the configuration file that are to be changed.
   

   { "appName":"MegaApp", "fileName":"Web.config", "changes":[ { "path":"configurations/navigation/sections/add", "filter":"name=OtherApp", "target":"link", "sourceName":"LinkToOtherApp" }, { "path":"configurations/connections/add", "filter":"name=MainDB", "target":"ConnectionString", "sourceName":"MainDBConnectionString" } ] } 

   
   appName - the name of the application;
   fileName - the name of the application configuration file, which we will change;
   changes - an array of changes that must be made in the configuration file when moving an application from one stand to another;
   path - path to the tag in the configuration file. (In my case, the configuration file has the xml format);
   filter - used when there are several identical tags along the specified path, and we need to take a specific one. We can filter it by the value of any parameter;
   target - tag parameter that we will change;
   sourceName is a certain alias by which we will determine the value substituted from the stand settings file.
   
   
   Sample Web.config File

    <?xml version="1.0" encoding="utf-8"?> <configuration> <navigation> <sections> <add name="OtherApp" text="1" link="http://OtherApp_dev.com" /> <add name="NotChangeApp" text="" link="http://habrahabr.ru" /> </sections> </navigation> <connectionStrings> <add name="MainDb" connectionString="data source=maindb-server;Initial Catalog=MAINDB;User ID=user;Password=pass" providerName="System.Data.SqlClient" /> <add name="NotChangeDb" connectionString="data source=localhost;Initial Catalog=DB;User ID=user;Password=pass;" providerName="System.Data.SqlClient" /> </connectionStrings> </configuration> 

   
   
   
   
3. Now about the source of the parameters of the stand. In my decision, I can use three different types of values ​​as the second parameter of the Configuration.ps1 script. The first is the path to the stand-up json-file, the second is the URL to the configuration storage web service (ConfigStorage), which returns a json-object, and the third is the path to the registry branch on the local machine which contains the URL to the ConfigStorage. I solved two tasks with the registry focus: first, I didn’t want to store the service URL on the CI server, since the settings of which stand will return the service depend on the parameters of the GET request. And the second - I wanted the uniformity of commands on the CI server, that is, the target server itself would determine “how to configure”, and not the continuous integration server.
4. The configuration storage service (ConfigStorage) I wrote in Python. Its essence is very simple: there are json-parameters of the stands, which are placed in a specific directory:
   

    /ConfigStorage /jsons /stand_dev.json /stand_stage.json /stand_prod.json 

   
   There is a service configuration file:
   

    { "keys_storage":[ { "key":"secret_dev", "pathToFile":"/jsons/stand_dev.json " }, { "key":"secret_stage", "pathToFile":"/jsons/stand_stage.json " }, { "key":"secret_prod", "pathToFile":"/jsons/stand_prod.json " } ] } 

   
   Accordingly, in order to get the data of a file, you need to send a GET request with the necessary key:
   

    http://ConfigStorage/ConfigStorage.py?key=secret_stage 

   
   In my case, security is achieved due to the closed network segment and domain policy, so I did not bother with encryption, although if there is such a need, it can also be realized.
   
   The stand parameter file looks like this:
   

    { "stand":"Stage", "settings":[ { "appName":"default", "sources":[ { "name":"LinkToOtherApp", "value":"http://OtherApp_stage.com" }, { "name":"MainDBConnectionString", "value":"data source=maindb-stage-server;Initial Catalog=MAINDB;User ID=user;Password=pass" } ] }, { "appName":"MegaApp2", "sources":[ { "name":"LinkToOtherApp", "value":"http://OtherApp_stage2.com" } ] } ] } 

   
   stand - the name of the stand;
   settings - an array of settings;
   appName is the name of the application. Here the meaning is this: by default, all the application configuration files are substituted for the values ​​from the section where appName is “default”, but if we need a different value for a particular application, then we create an additional section with appName, where we override this value. In my example for the MegaApp application, the value of

     http://OtherApp_stage.com 

   , and for MegaApp2 - value

     http://OtherApp_stage2.com. 

   sources - an array of aliases ( name ) and their values ​​( value )
   
5. And the last step in the configuration file of the Web.config application is substituted values ​​according to the change file Webconfig.json from the stand parameters file stand_stage.json.
6. Profit.

Why is all this necessary?

Here is what it gives:

• You have a complete and current description of which application parameters change depending on the stand;
• You have a complete and current description of all parameters of all stands;
• Since these descriptions are in the version control system, you get a complete history of the changes and the reasons for these changes;
• You get a single point of all changes, you need to make changes in only one place;
• You get a guarantee that all configuration files on all stands are the same. The developers' favorite excuse, “it works on my computer, it means you have done something wrong” stops working;
• The non-technical character is that there is a clear line of responsibility between the parties. The developers are responsible for the correctness of the configuration file and the change file, and the engineers for the correctness of the bench parameter files. The watershed runs at the repository level.
• Testers for their experiences can create the most perverted stand parameter files and instantly switch all applications to this or that configuration;
• Of course, you will get the greatest pleasure when there are many different applications in your zoo that are somehow related to each other. I still stretch in a contented smile, when the whole gang of friendly together takes the position I need in one click.

PS: Unfortunately, I can’t put the source code, as they are the property of the company, but I spent much more time writing the article than writing the scripts themselves.