Optimize the build of a large project
The problem of increasing the project build time is faced by almost all developers at least once a year. Assembly is not a quick job, which is especially unpleasant when using the practice of Continuous Integration with its constant reassembly and related activities. A lengthy build negates all the benefits of continuous integration, and a simple increase in computing power does not always produce the desired effect.
In the process of developing and further developing a large heterogeneous project, we also faced optimization problems. But there must be ways to reduce build time? We decided to find a way to solve this problem. Collect in 60 seconds! ..
')
About what eventually managed to achieve, and will be discussed in our topic.
The key to successful project optimization is its automation. We used the well-known automation system CMake, on which, as on the foundation of a house, various technologies were built on.
Here is what we had at the beginning of the journey:
All this was going for 12 minutes.
After optimizing the code based on the results of static analysis, removing unnecessary #include directives, rebuilding executable modules, dynamic and static libraries, the build time was reduced to 8 minutes. The following discussion focuses on optimizing the build process, when all possible code improvements have already been implemented.
To achieve maximum speed, we used two approaches to optimization: hardware and software. As we show below, the first does not exclude the second at all, but complements it.
About typical customization methods VisualStudio is written a lot on the Internet, so let's touch them casually.
More interesting is the technology known as UnityBuild . This is one of the most effective software optimization methods, accelerating the assembly by 30%. In the course of work, we slightly improved this technology and called a new, improved version of Smart UnityBuild.
The standard technology works as follows: in each * .cmake file of each project, the module unity_build.cmake is used, which is directed to the folder containing all * .cpp files, processes them and combines them into a new “common” file * .cpp. Next, a * .vcproj file is created that contains this single * .cpp.
Our option does not work quite like that. In some cases (in particular, when working with a large project), creating one common * .cpp-file is not the best option. Therefore, we decided to generate several files instead of one large one.
Schematically it looks like this:
As a result, the use of our technology Smart UnityBuild allowed us to speed up the assembly to 4 minutes. 40 seconds
It is obvious that the use of PCH in conjunction with the UnityBuild technology, which is called in the forehead, will not give anything. But still there is a way to trick VisualStudio. To do this, you need to artificially bring the files used in the project several times (we have * .boost and * .spl) into a separate project, assemble Precompiled Headers from it and put them into the main project, as a result VisualStudio will “think” that these are real PCH. When using Unity Build technology with Precompiled Headers and the LP key, you can achieve a significant reduction in build time: in our case, from 12 minutes. up to 4 min. 38 seconds (without LP key - 5 min. 53 sec.).
Reduce the time to build a project can, of course, not only by combining different technologies. The process can also be significantly accelerated by optimizing the operation of server hardware.
The best result in terms of assembly time was achieved with the use of 7 SSD drives in the Raid0 array and RAMDisk: the duration of the project assembly was about 6 minutes.
The increase in speed more than doubled compared with the standard time value (7 min. 45 sec.) Gives the use of 32 processors with Hyper Threading technology (3 min. 22 sec.).
By combining all means from Visual Studio options to using SSD drives, we were able to reduce the project build time from 12 minutes to 1 minute. 45 seconds Thus, the stated goal - to keep within 60 seconds - remained unfulfilled, but we were not upset :) Reducing the time spent on assembling a project by more than 6 times is a good result.
Here is the progress of the assembly of our project:
From the diagram it is clear that several time-consuming processes interfere with the achievement of even greater speed. This is a linkage of large modules, which cannot be parallelized (only a partial attempt to do this in MSVS 2012), but you can programmatically break them into several small ones, reduce the number and size of static libraries.
That's all. We did not have a goal to embrace the immense, and some technical aspects of optimization were not included in the article. We are ready to reveal the details and answer all the questions (including cmake source code) in the comments. Thank you for your attention!
PS The thoughts expressed in this topic formed the basis for the presentation, which Viktor Strelkov (Head of Research and Quality Control at Positive Technologies) presented at the CEE-SECR conference in November 2012. A recording of the broadcast of this speech can be viewed on the event website .
In the process of developing and further developing a large heterogeneous project, we also faced optimization problems. But there must be ways to reduce build time? We decided to find a way to solve this problem. Collect in 60 seconds! ..
')
About what eventually managed to achieve, and will be discussed in our topic.
Start
The key to successful project optimization is its automation. We used the well-known automation system CMake, on which, as on the foundation of a house, various technologies were built on.
Here is what we had at the beginning of the journey:
- 176 projects of C ++ and C # (in a ratio of approximately 60% to 40%);
- 100 MB of sources;
- 18,000 files (> 3 million lines);
- MSVS;
- server (8 CPU, normal disk on 10 HDD roads, 12 GB of RAM);
- virtual machine.
All this was going for 12 minutes.
After optimizing the code based on the results of static analysis, removing unnecessary #include directives, rebuilding executable modules, dynamic and static libraries, the build time was reduced to 8 minutes. The following discussion focuses on optimizing the build process, when all possible code improvements have already been implemented.
To achieve maximum speed, we used two approaches to optimization: hardware and software. As we show below, the first does not exclude the second at all, but complements it.
Program method
Typical methods
About typical customization methods VisualStudio is written a lot on the Internet, so let's touch them casually.
- Key / MP, which allows you to run multiple compilation processes simultaneously.
- The / Z7 key, which disables the generation of debug information, resulting in compilation is faster.
More interesting is the technology known as UnityBuild . This is one of the most effective software optimization methods, accelerating the assembly by 30%. In the course of work, we slightly improved this technology and called a new, improved version of Smart UnityBuild.
The standard technology works as follows: in each * .cmake file of each project, the module unity_build.cmake is used, which is directed to the folder containing all * .cpp files, processes them and combines them into a new “common” file * .cpp. Next, a * .vcproj file is created that contains this single * .cpp.
Our option does not work quite like that. In some cases (in particular, when working with a large project), creating one common * .cpp-file is not the best option. Therefore, we decided to generate several files instead of one large one.
Schematically it looks like this:
As a result, the use of our technology Smart UnityBuild allowed us to speed up the assembly to 4 minutes. 40 seconds
UnityBuild + PCH (Incredibuild)
It is obvious that the use of PCH in conjunction with the UnityBuild technology, which is called in the forehead, will not give anything. But still there is a way to trick VisualStudio. To do this, you need to artificially bring the files used in the project several times (we have * .boost and * .spl) into a separate project, assemble Precompiled Headers from it and put them into the main project, as a result VisualStudio will “think” that these are real PCH. When using Unity Build technology with Precompiled Headers and the LP key, you can achieve a significant reduction in build time: in our case, from 12 minutes. up to 4 min. 38 seconds (without LP key - 5 min. 53 sec.).
Hardware method
Reduce the time to build a project can, of course, not only by combining different technologies. The process can also be significantly accelerated by optimizing the operation of server hardware.
SSD RAID0 x 7
The best result in terms of assembly time was achieved with the use of 7 SSD drives in the Raid0 array and RAMDisk: the duration of the project assembly was about 6 minutes.
32 CPU x 3 GHz
The increase in speed more than doubled compared with the standard time value (7 min. 45 sec.) Gives the use of 32 processors with Hyper Threading technology (3 min. 22 sec.).
Total
By combining all means from Visual Studio options to using SSD drives, we were able to reduce the project build time from 12 minutes to 1 minute. 45 seconds Thus, the stated goal - to keep within 60 seconds - remained unfulfilled, but we were not upset :) Reducing the time spent on assembling a project by more than 6 times is a good result.
Here is the progress of the assembly of our project:
From the diagram it is clear that several time-consuming processes interfere with the achievement of even greater speed. This is a linkage of large modules, which cannot be parallelized (only a partial attempt to do this in MSVS 2012), but you can programmatically break them into several small ones, reduce the number and size of static libraries.
That's all. We did not have a goal to embrace the immense, and some technical aspects of optimization were not included in the article. We are ready to reveal the details and answer all the questions (including cmake source code) in the comments. Thank you for your attention!
PS The thoughts expressed in this topic formed the basis for the presentation, which Viktor Strelkov (Head of Research and Quality Control at Positive Technologies) presented at the CEE-SECR conference in November 2012. A recording of the broadcast of this speech can be viewed on the event website .
Source: https://habr.com/ru/post/171801/
All Articles