Personal experience: TeamCity and character server integration

We continue to talk about how to implement practices and approaches DevOps. For example, they recently wrote about creating their own tool for conducting competitive analysis of software products. Today we will talk about how we integrated the TeamCity system with an external debug symbols server.

Why do you need it

Debug symbols are information that the compiler automatically generates based on source codes. Such symbols can be included in the binary file, presented as a separate file, or discarded altogether at the compilation stage - this allows reducing the size of the binary file and increasing the complexity of its reverse engineering.
')
Such symbols solve several problems:

• they contain debug information about the binary file (the names of variables, functions, etc.)
• with their help it is easier to look for errors in the code;
• debugging symbols are needed when searching for bugs and parsing dumps.

The use of debug symbols is clear, now let's talk about why we did not use the symbol server built into TeamCity and came to a different solution.

What is wrong with the TeamCity symbol server

Our decision to use an external character server was dictated by the flaws of the built-in TeamCity solution:

• At the time when we were thinking about how to work with debugging symbols, the built-in solution TeamCity did not know how to interact with binary files - this function appeared later, but we decided to use an external server.
• Also, there was no debugged character storage mechanism.
• In the configuration we had at that time, there was not a large amount of disk resources on the character server; we simply did not have enough space.
• Our company is engaged in the development of multi-component products, and it was difficult to store symbols with version binding components for them.

Decision

Here is how we solve the problems described now:

• use a virtual server with OS Windows Server;
• The main working tool is included in the Windows SDK Symstore.exe, which is used to lay out characters on the server;
• We also wrote a separate metaranner for TeamCity.

We have implemented a so-called “release scheme with assembly promotions” in which the tested artifacts of assemblies of individual components are moved from the garbage .snapshot repository to the release version - from there they are later taken to build release installers of products. We lay out the symbols of release and feature assemblies on different shared-servers. Therefore it is necessary to be able to distinguish release assemblies. When we build a regular build, the binary files go to the artifactories, like the debug symbols. When you start the promotion of the assembly of a separate component through a special configuration — the promoter, the developed metainner loads symbols into the release folder on the symbol server.



In more detail, this process looks like this: when an assembly occurs, we look at whether it has the status of promote. If we are dealing with such an assembly, the characters are placed in the release shared folder on the server. If this is a common feature, the characters are placed in the artifactories and in the common “development” folder on the character server.

What is the result

As a result, two directories are now located on our symbol server:

• Release Symbols - there are symbols of release assemblies in it that need to be stored for a long time;
• Symbols - for symbols of developer assemblies.

We managed to solve the problem with a lack of disk space - a directory with developer characters is cleared by a script that is launched by the scheduler. It deletes all files older than N days from a certain directory of the tree of characters, as well as files that meet a certain condition — for example, all assemblies of the same version, except the latest one.

Our TeamCity metaranner is able to work with all binary files and is integrated into the process of building multi-component products.

PS The story about our experience with TeamCity and the character server was presented in the framework of DevOps-mitap, which took place in the fall of 2016 in Moscow.

Video:



Slides



The link presents presentations of 16 reports presented during the event. All presentations and video presentations are added to the table at the end of this topic-announcement .

Author : Alexey Soloviev