Using, configuring and testing distcc and ccache

Prehistory:

Once I got the idea to test the effectiveness of tools such as distcc and ccache. They were used for a long time, but the thought about the expediency of all this was still spinning in my head. And finally, the hands came to check in practice, I immediately warn you that everything was done for yourself, in real conditions, and the results of your tests may not coincide with those given below. Consider installing and configuring these components, and then testing the assembly speed with the time command.

Theory:

To begin with, we will define what it is and why it may be useful. Although if you are reading this, you probably already understand why these tools are used.
A few words about Distcc from Wikipedia.
Distcc (distributed C / C ++ / ObjC compiler) is a tool that allows you to compile source codes using C / C ++ / ObjC compilers on remote machines, which allows you to speed up the compilation process. It consists of two main parts - server (distccd) and client (distcc).
Ccache (from the English compiler cache; pronounced "sekesh") - cache compilers for C and C ++ for Linux and other Unix-like systems.
Using csache can significantly speed up the build of some packages or projects that are compiled several times, since only files that have changed since the last compilation will be recompiled.

What do we do?

1. Install and configure distcc and ccache.
2. Let's check how effectively these mechanisms work together and separately.

The following infrastructure is available: 100 Mib / s network, with the following machines:

	Distcc client	Distcc server	Distcc server	Distcc server
CPU	Intel Pentium Dual E2160 1.80GHz	Intel Xeon 3.20GHz	Intel Pentium D CPU 3.00GHz	Intel Pentium 4 CPU 3.00GHz
Memory	1 GB	1 GB	1 GB	1 GB
Gcc	4.4.5	4.4.5	4.4.5	4.4.5
OS	gentoo x86	gentoo x86	debian 5.0.7 x86	debian 5.0.7 x86

Apparently the machines are rather weak, but they are coping with their tasks, at the time of the tests, all third-party services were turned off. Also, as you can see, all machines have one version of GCC installed (it is recommended to use the same version, a difference in the last digit of version 4.4. * Is allowed), but earlier some machines also used GCC 4.4.3 and everything worked fine.
')

Installation and configuration:

First you need to install and configure our tools.

Gentoo

The installation process of distcc and ccache is described in detail in the manual , I will only briefly go through the main stages, I will show some nuances.
Install distcc:
emerge -avt distcc
We look at the USE flags, that we need to leave, that we do not need to remove and start the installation.
After the distcc is assembled and installed, the distccd daemon parameters will be in the /etc/conf.d/distccd file, the list of used hosts in / etc / distcc / hosts. In /etc/conf.d/distccd, you need to make changes to the parameters of the distccd server.
Add the possibility of logging, it is necessary for debugging, in the future, you can disable:
DISTCCD_OPTS="${DISTCCD_OPTS} --log-file /var/log/distcc.log --log-level info "
We define the list of hosts (networks) whose requests we will process:
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
Specify the interface and port on which the daemon will be launched:
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.0.174"
It is also possible to specify the priority with which the service will be launched, this value must be set strictly individually, by default its value is 15:
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"
It is necessary to tell the system that you can use distcc, add to make.conf:
FEATURES="distcc"
And we define the directory for temporary distcc files:
DISTCC_DIR="/tmp/.distcc"
For those who do not mind the memory and want to increase performance, you can take out a directory of temporary files on tmpfs (the file system is organized in virtual memory (RAM + swap). Suitable for storing temporary files.)
mount -t tmpfs tmpfs -o size=850M /tmp/
If tmpfs overflows, data will be written to the swap, which may adversely affect the performance of other programs.
For permanent use, add this line to / etc / fstab:
tmpfs /tmp tmpfs size=850M,mode=0777 0 0

Next, you need to start the service itself:
/etc/init.d/distccd start
And you can also add it to autoload:
rc-update add distccd default

I will remind these changes we made for the server part of the distcc.
Now let's see what needs to be done on the client side. Everything is simple, we need to add the list of hosts that act as distcc servers to the / etc / distcc / hosts file in one line separated by a space (you can specify both machine names and their ip addresses). Enumeration of machines is recommended to begin in order of decreasing power. A client can be a server, and vice versa.

Install ccache by manual
emerge -avt ccache
We look at the USE flags, that we need to leave, that we do not need to remove and start the installation.
After ccache is assembled and installed, we make changes to /etc/make.conf. Let us tell the system that it is necessary to use ccache:
FEATURES="distcc ccache "
Set the cache size:
CCACHE_SIZE="2G"
Determine the place where the cache will be located:
CCACHE_DIR="/var/tmp/ccache"
The cache cache can also be stored in tmpfs:
mount -t tmpfs tmpfs -o size=2G /var/tmp/ccache
or register in fstab:
tmpfs /var/tmp/ccache tmpfs size=2G,mode=0777 0 0
However, in the case of a system reboot, all data in tmpfs will be lost, which will nullify all the ccache developments.

Debian

Install distcc. Since the client will be a machine with gentoo on board, then on machines with a Debian OS, you will have to tweak something.
Install distcc and ccache:
apt-get install distcc
apt-get install ccache
The distccd daemon settings are in the / etc / default / distcc file, the list of used hosts is in / etc / distcc / hosts.
Make changes to / etc / default / distcc. Enable daemon startup when the machine boots:
STARTDISTCC="true"
Specify the subnet with which we listen to requests:
ALLOWEDNETS="192.168.3.0/24"
Specify the address at which the service will listen to the port:
LISTENER="192.168.3.103"
Priority with which the running process will work:
NICE="15"
Enable or detect via mDNS / DNS-SD
ZEROCONF="false"
The values ​​in / etc / distcc / hosts are also filled in as needed by gentoo (I have debian only used as a distcc server).
Now we need to add a symbolic link to the gcc version used, the link should have the name of the executable file on the client (i686-pc-linux-gnu-c ++):
cd /usr/bin/
ln -s ./g++-4.4 ./i686-pc-linux-gnu-c++
ln -s ./cpp-4.4 ./i686-pc-linux-gnu-cpp
ln -s ./gcc-4.4 ./i686-pc-linux-gnu-gcc
ln -s ./g++-4.4 ./i686-pc-linux-gnu-g++
Run the distcc server:
/etc/init.d/distccd start
It should be clarified that links like i686-pc-linux-gnu-c ++ should be created, because a hard link with the same name, which is also gcc-4.4, is launched on the client system (gentoo) when compiling.
Also, for best results, I placed the build directory for temporary files in the tmpfs file system.

Testing

Now let's try to check the effectiveness of the examples. For the purity of the experiment, the assembly was carried out 3 times. We will collect mplayer, just the first one got caught. Testing will be carried out using the time utility, which yields the following indicators:
user is the total number of seconds of CPU time spent in user mode.
sys is the total number of seconds of CPU time spent in kernel mode.
real - actually elapsed time (in seconds).

1 Building mplayer on a local host without using distcc and ccache (-distcc -ccache) with the -j2 option
time MAKEOPTS="-j2" FEATURES="-distcc -ccache" emerge -vt mplayer

one	2	3
real 3m56.979s user 6m11.729s sys 0m44.551s	real 3m55.825s user 6m12.151s sys 0m43.660s	real 3m55.745s user 6m12.351s sys 0m43.671s

2 Building mplayer on a local host without using distcc and ccache (-distcc -ccache) with the -j3 option
time MAKEOPTS="-j3" FEATURES="-distcc -ccache" emerge -vt mplayer

one	2	3
real 3m57.662s user 6m15.214s sys 0m43.392s	real 3m58.108s user 6m14.779s sys 0m43.656s	real 3m57.901s user 6m14.894s sys 0m43.464s

3 Building mplayer on a local host without using distcc, but with ccache enabled (-distcc ccache) with the -j2 option
time MAKEOPTS="-j2" FEATURES="-distcc ccache" emerge -vt mplayer

one	2	3
real 4m14.438s user 6m28.199s sys 1m0.705s	real 0m58.587s user 0m39.101s sys 0m25.706s	real 0m57.901s user 0m38.940s sys 0m25.564s

4 Building mplayer on a local host using distcc, but without ccache (distcc -ccache) with the -j10 option
time MAKEOPTS="-j10" FEATURES="distcc -ccache" emerge -vt mplayer

one	2	3
real 2m26.516s user 1m6.692s sys 0m34.821s	real 2m27.065s user 1m6.643s sys 0m34.856s	real 2m26.965s user 1m6.593s sys 0m34.569s

5 Building mplayer on a local host using distcc and ccache (distcc ccache) with the -j10 option
time MAKEOPTS="-j10" FEATURES="distcc ccache" emerge -vt mplayer

one	2	3
real 2m28.590s user 1m16.117s sys 0m45.723s	real 0m57.456s user 0m39.892s sys 0m24.781s	real 0m57.411s user 0m39.947s sys 0m24.692s

I decided to try something harder, for example chromium

5 Putting it first on a local machine without distcc and ccache.
time MAKEOPTS="-j2" FEATURES="-distcc -ccache" emerge -vt chromium

one	2	3
real 67m30.435s user 116m58.378s sys 11m57.523s	real 66m50.345s user 115m48.389s sys 11m59.223s	real 67m45.165s user 116m59.548s sys 11m57.530s

6 Let's try building using distcc (distcc -ccache)
time MAKEOPTS="-j10" FEATURES="distcc -ccache" emerge -vt chromium

one	2	3
real 36m25.392s user 16m2.063s sys 8m8.307s	real 35m15.291s user 15m10.604s sys 7m4.378s	real 35m22.390s user 15m1.423s sys 7m5.156s

7 Already better, now try the same thing, but using distcc and ccache.
time MAKEOPTS="-j10" FEATURES="distcc ccache" emerge -vt chromium

one	2	3
real 32m2.656s user 18m56.006s sys 10m57.023s	real 17m22.228s user 18m1.757s sys 9m36.169s	real 16m59.284s user 17m59.577s sys 8m35.679s

Conclusion:

Well, you can look at the whole picture, and summarize the results of the tests. And the conclusion is the following: the use of distcc reduces the build time by about two times, and when paired with ccache, the speed increases by about four times. For myself, I decided that I will continue to use these tools.