TL; DR . In this article, we explore hardening schemes that work out of the box in five popular Linux distributions. For each, we took the default kernel configuration, loaded all the packages, and analyzed the protection schemes in the nested binary files. OpenSUSE 12.4, Debian 9, CentOS, RHEL 6.10 and 7 distributions, as well as Ubuntu 14.04, 12.04 and 18.04 LTS are considered.

The results confirm that even basic schemes, such as stack canaries and position-independent code, are not yet used by everyone. The situation is even worse for compilers when it comes to protecting against vulnerabilities like stack clash, which hit the spotlight in January after publishing vulnerability information in systemd . But not everything is so hopeless. In large part of the binaries, basic protection methods are implemented, and their number is growing from version to version.

The audit showed that the largest number of protection methods is implemented in Ubuntu 18.04 at the OS and application level, followed by Debian 9. On the other hand, in OpenSUSE 12.4, CentOS 7 and RHEL 7 the basic protection schemes are also implemented, and stack collision protection is applied even more widely with a much tighter set of packages by default.

Introduction

It is difficult to provide high quality software. Despite the huge number of advanced tools for static code analysis and dynamic analysis at runtime, as well as significant progress in the development of compilers and programming languages, modern software still suffers from vulnerabilities that are constantly exploited by hackers. The situation is even worse in ecosystems that include outdated code. In such cases, we are not only confronted with the eternal problem of finding possible exploitable errors, but are also limited by rigid backward compatibility frameworks, which often require maintaining a limited, and even worse, vulnerable or buggy code.
')
This is where hardening comes into play. We cannot prevent some types of errors, but we can make the attacker's life more difficult and partially solve the problem by preventing or preventing the operation of these errors. Such protection is used in all modern operating systems, but the methods vary greatly in complexity, efficiency and performance: from stack canaries (stack canaries) and ASLR to full-fledged protection CFI and ROP . In this article, we will look at what protection methods are used in the most popular Linux distributions in the default configuration, and also examine the properties of binaries that are distributed through the package management systems of each distribution.

CVE and security

We all have seen articles with titles like “The Most Vulnerable Applications of the Year” or “The Most Vulnerable Operating Systems.” Usually, statistics on the total number of records of vulnerabilities such as CVE (Common Vulnerability and Exposures) obtained from the National Vulnerability Database (NVD) from NIST and other sources are given there. Subsequently, these applications or operating systems are ranked by the number of CVEs. Unfortunately, although CVE is very useful in tracking down problems and informing vendors and users, they say little about the real security of software.

For example, consider the total number of CVEs over the past four years for the Linux kernel and the five most popular server distributions, namely, Ubuntu, Debian, Red Hat Enterprise Linux, and OpenSUSE.


Fig. one

What does this chart tell us? Does a greater number of CVEs mean that one distribution is more vulnerable than another? No answer. For example, in this article you will see that Debian implements tougher protection mechanisms compared to, say, OpenSUSE or RedHat Linux, and yet Debian has more CVE. However, they do not necessarily mean weakened security: even the presence of a CVE does not indicate whether the vulnerability is exploitable . The severity scores give an idea of ​​how likely the exploit is to exploit, but ultimately exploitation depends to a large extent on the protection present in the affected systems, as well as on the resources and capabilities of the intruders. Moreover, the lack of CVE reports says nothing about other unregistered or unknown vulnerabilities. The difference in CVE can be explained not by software quality, but by other factors, including the resources allocated for testing, or the size of the user base. In our example, more CVE from Debian may simply indicate that Debian ships more software packages.

Of course, the CVE system provides useful information that allows you to create appropriate protections. The better we understand the causes of a program crash, the easier it is to identify possible methods of operation and develop appropriate detection and response mechanisms. In fig. 2 shows the categories of vulnerabilities for all distributions over the past four years ( source ). It is immediately obvious that most CVEs fall into the following categories: denial of service (DoS), code execution, overflow, memory corruption, information leakage (exfiltration), and privilege escalation. Although many CVEs are counted several times in different categories, in general, the same problems persist year after year. In the next part of the article, we will evaluate the use of various protection schemes to prevent the exploitation of these vulnerabilities.


Fig. 2

Tasks

In this article we intend to answer the following questions:

• What is the security of various Linux distributions? What security mechanisms exist in the kernel and user-space applications?
  
• How has the adoption of protection mechanisms for different distributions changed over time?
  
• What are the average dependencies of packages and libraries for each distribution?
  
• What protections are implemented for each binary?

The choice of distros

It turns out that it is difficult to find accurate statistics on the installations of distributions, since in most cases the number of downloads does not indicate the number of real installations. However, Unix variants constitute the majority of server systems (69.2% on web servers, according to statistics from W3techs and other sources), and their share is constantly growing. Thus, for our research, we focused on the distributions available out of the box on the Google Cloud platform. In particular, we have chosen the following OS:

Distribution / Version	Core	Bild
OpenSUSE 12.4	4.12.14-95.3-default	# 1 SMP Wed Dec 5 06:00:48 UTC 2018 (63a8d29)
Debian 9 (stretch)	4.9.0-8-amd64	# 1 SMP Debian 4.9.130-2 (2018-10-27)
CentOS 6.10	2.6.32-754.10.1.el6.x86_64	# 1 SMP Tue Jan 15 17:07:28 UTC 2019
CentOS 7	3.10.0-957.5.1.el7.x86_64	# 1 SMP Fri Feb 1 14:54:57 UTC 2019
Red Hat Enterprise Linux Server 6.10 (Santiago)	2.6.32-754.9.1.el6.x86_64	# 1 SMP Wed Nov 21 15:08:21 EST 2018
Red Hat Enterprise Linux Server 7.6 (Maipo)	3.10.0-957.1.3.el7.x86_64	# 1 SMP Thu Nov 15 17:36:42 UTC 2018
Ubuntu 14.04 (Trusty Tahr)	4.4.0–140-generic	# 166 ~ 14.04.1-Ubuntu SMP Sat Nov 17 01:52:43 UTC 20 ...
Ubuntu 16.04 (Xenial Xerus)	4.15.0–1026-gcp	# 27 ~ 16.04.1-Ubuntu SMP Fri Dec 7 09:59:47 UTC 2018
Ubuntu 18.04 (Bionic Beaver)	4.15.0–1026-gcp	# 27-Ubuntu SMP Thu Dec 6 18:27:01 UTC 2018

Table 1

Analysis

Let us study the default kernel configuration, as well as the properties of the packages available through the package manager of each distribution kit out of the box. Thus, we consider only packages from the default mirrors of each distribution, ignoring packages from unstable repositories (for example, 'testing' mirrors in Debian) and third-party packages (for example, Nvidia packages from standard mirrors). In addition, we do not consider custom kernel compilations or configurations with enhanced security.

Kernel configuration analysis

We applied the analysis script based on the free kconfig checker . Consider the protection settings out of the box for the named distributions and compare them with the list from the Kernel Self-Defense Project (KSPP). For each configuration parameter, table 2 describes the desired setting: there is a check mark for distributions that correspond to KSSP recommendations (for an explanation of the terms, see here ; in future articles we will describe how many of these protection methods appeared and how to hack the system in their absence).





In general, the new kernels are more stringent settings out of the box. For example, CentOS 6.10 and RHEL 6.10 on the 2.6.32 kernel do not have most of the critical functions implemented in new kernels, such as SMAP , strict RWX permissions, address randomization, or copy2usr protection. It should be noted that many of the configuration options from the table are missing in older kernel versions and are not applicable in reality - in the table this is still indicated as a lack of adequate protection. Similarly, if the configuration parameter is not present in this version, and for security, this parameter must be disabled, this is considered a reasonable configuration.

Another point in interpreting the results: some kernel configurations that increase the attack surface can also be used for security. Examples include uprobes and kprobes, kernel modules and BPF / eBPF. Our recommendation is to use the above mechanisms to provide real protection, since they are non-trivial to use, and their operation assumes that the malicious actors are already entrenched in the system. But if these options are enabled, the system administrator should actively monitor abuse.

Studying further the entries in Table 2, we see that modern kernels provide several options for protecting exploitation of such vulnerabilities as information leakage and stack / heap overflow. However, we notice that even the most recent popular distributions have not yet implemented more complex protection (for example, with grsecurity patches) or modern protection against code reuse attacks (for example, a combination of randomization with R ^ X schemes for code ). Worse, even these more advanced defenses do not protect against the full range of attacks. Thus, it is extremely important for system administrators to complement intelligent configurations with solutions that offer detection and prevention of exploits at runtime.

Application analysis

Not surprisingly, different distributions have different package characteristics, compilation options, library dependencies, etc. Differences exist even for related distributions and packages with a small number of dependencies (for example, coreutils in Ubuntu or Debian). To assess the differences, we downloaded all the available packages, extracted their contents, and analyzed the binary files and dependencies. For each package, we tracked other packages on which it depends, and for each binaries we tracked its dependencies. This section summarizes the findings.

Distributions

In total, we downloaded 361,556 packages for all distributions, extracting only packages from mirrors by default. We ignored packages without ELF executable files, such as source codes, fonts, etc. After filtering, 129,569 packages remain, containing a total of 584,457 binary files. Distribution of packages and files by distributions is shown in fig. 3


Fig. 3

It can be noted that the more modern the distribution kit is, the more packages and binary files it contains, which is logical. At the same time, Ubuntu and Debian packages include much more binary files (both executable and dynamic modules and libraries) than CentOS, SUSE and RHEL, which potentially affects the attack surface of Ubuntu and Debian (it should be noted that the numbers reflect all binary versions of all package, that is, some files are analyzed several times). This is especially important when considering the dependencies between packages. Thus, the vulnerability in the binary of one package can affect many parts of the ecosystem, as a vulnerable library can affect all binary files importing it. As a point of reference, we will look at the distribution of the number of dependencies among packages in different operating systems:


Fig. four

In almost all distributions, 60% of packages have at least 10 dependencies. In addition, for some packages, the number of dependencies is much larger (more than 100). The same applies to inverse dependencies of packages: as expected, several packages are used by many other packages in the distribution, therefore vulnerabilities in these few selected people are at high risk. As an example, the following table lists the 20 packages with the maximum number of inverse dependencies in SLES, Centos 7, Debian 9, and Ubuntu 18.04 (each cell indicates the package and the number of inverse dependencies).


Table 3

Interesting fact. Although all analyzed OSs are built for x86_64 architecture, and for most packages, the architecture is defined as x86_64 and x86, but packages often contain binary files for other architectures, as shown in fig. five.


Fig. five

In the next section, we delve into the characteristics of the binaries being analyzed.

Binary file protection statistics

As an absolute minimum, you need to examine the basic set of protection options for existing binary files. Several Linux distributions come with scripts that perform such checks. For example, in Debian / Ubuntu there is such a script. Here is an example of his work:

$ hardening-check $(which docker) /usr/bin/docker: Position Independent Executable: yes Stack protected: yes Fortify Source functions: no, only unprotected functions found! Read-only relocations: yes Immediate binding: yes 

The script checks five security functions :

• Position Independent Executable (PIE): indicates whether the text section of the program can be moved into memory to achieve randomization if ASLR is enabled in the kernel.
  
• Stack Protected: whether stack canaries are included to protect against stack collision attacks.
  
• Fortify Source: whether unsafe functions (for example, strcpy) are replaced by their more secure counterparts, and calls tested in runtime are replaced by their counterpartable ones (for example, memcpy instead of __memcpy_chk).
  
• Read-only relocations (RELRO): Whether the records in the relocation table are marked as read-only if they have worked before execution.
  
• Immediate binding: Does the runtime binder allow all moves before starting the program (this is equivalent to a full RELRO).

Are the above mechanisms sufficient? Unfortunately not. There are ways to circumvent all of the above defenses, but the more stringent the defense, the higher the bar for the attacker. For example, RELRO traversal methods are more difficult to apply if PIE and immediate binding are in effect. Similarly, a full ASLR requires additional work to create a working exploit. However, sophisticated attackers are ready to meet such defenses: their absence will in fact speed up hacking. Therefore, it is imperative that these measures are considered as a necessary minimum .

We wanted to study how many binary files in the distributions in question are protected by these, as well as three more methods:

• A non-executable bit ( NX ) prevents execution in any region that should not be executable, for example, on a stack heap, etc.
• RPATH / RUNPATH indicates the execution path used by the dynamic loader to search for relevant libraries. The first is mandatory for any modern system: its absence allows attackers to write down the payload in memory and execute it as it is. For the second, incorrect configurations of the execution path help in the introduction of an unreliable code, which can lead to a number of problems (for example, escalation of privileges , as well as other problems ).
• Stack collision protection provides protection against attacks that cause the stack to overlap with other areas of memory (for example, a heap). Given the recent exploits abusing heap collision vulnerabilities in systemd , we found it appropriate to include this mechanism in our data set.

So, without further ado, let's go over the numbers. Tables 4 and 5 contain squeeze analysis of executable files and libraries of various distributions, respectively.

• As you can see, NX protection is implemented everywhere, with rare exceptions. In particular, it can be noted that its use is somewhat lower in Ubuntu and Debian distributions compared to CentOS, RHEL and OpenSUSE.
  
• Stacked canaries are absent in many places, especially in distributions with old kernels. Some progress has been observed in the latest Centos, RHEL, Debian and Ubuntu distributions.
  
• With the exception of Debian and Ubuntu 18.04, most distros have poor PIE support.
  
• Stack collision protection is poorly implemented in OpenSUSE, Centos 7 and RHEL 7 and is almost absent from the rest.
  
• All distros with modern kernels have some support for RELRO, with Ubuntu 18.04 leading, and Debian ranks second.

As already mentioned, the metrics in this table are averages for all versions of a binary file. If you only look at the latest versions of the files, the numbers will be different (for example, see Debian Progress using PIE ). Moreover, most distributions usually check the protection of only a few functions in binary code when calculating statistics, and our analysis shows the true percentage of fortified functions. Therefore, if 5 out of 50 functions are protected in a binary, we will give it a grade of 0.1, which corresponds to 10% of fortified functions.


Table 4. Protection characteristics for executable files shown in Fig. 3 (implementation of the relevant functions as a percentage of the total number of executable files)


Table 5. Protection characteristics for the libraries shown in Fig. 3 (implementation of the corresponding functions as a percentage of the total number of libraries)

So is there any progress? Definitely there: it can be seen from the statistics on individual distributions (for example, Debian ), as well as from the tables above. As an example in fig. 6 shows the introduction of defense mechanisms in three consecutive distributions of Ubuntu LTS 5 (we omitted the stack collision protection statistics). We notice that from version to version more and more files support stack canaries, and also successively more and more binary files come with full RELRO protection.


Fig. 6

Unfortunately, a number of executable files in different distributions still do not have any of the above protection. For example, looking at Ubuntu 18.04, you can see the ngetty binary (a replacement for getty), as well as the mksh and lksh shells, the picolisp interpreter, the nvidia-cuda-toolkit packages (a popular package for GPU-accelerated applications such as machine learning frameworks) and klibc -utils Similarly, the mandos-client binary (an administrative tool that automatically restarts machines with encrypted file systems), as well as rsh-redone-client (re-implementation of rsh and rlogin) comes without NX protection, although they have SUID rights :(. In addition, several suid binaries do not have basic protection, such as stack canaries (for example, the Xorg.wrap binary file from the Xorg package).

Summary and Concluding Remarks

In this article, we have identified several security properties of modern Linux distributions. The analysis showed that the latest Ubuntu LTS distribution (18.04) implemented on average the strongest protection of the OS level and applications among distributions with relatively new kernels, such as Ubuntu 14.04, 12.04 and Debian 9. However, the considered CentOS, RHEL and OpenSUSE distributions in our set Data defaults to a more dense set of packages, and in recent versions (CentOS and RHEL) have a higher percentage of stack collision protection implementations compared to competitors based on Debian (Debian and Ubuntu). Comparing the versions of CentOS and RedHat, we notice big improvements in the implementation of stack canaries and RELRO from versions 6 to 7, but on average CentOS has more functions than RHEL. In general, all distributions should pay particular attention to protecting PIE, which, with the exception of Debian 9 and Ubuntu 18.04, is implemented in less than 10% of the binary files from our data set.

Finally, it should be noted: although we conducted a manual survey, there are many security tools (for example, Lynis , Tiger , Hubble ) that perform the analysis and help avoid unsafe configurations. Unfortunately, even strong protection in reasonable configurations does not guarantee the absence of exploits. That is why we firmly believe that it is vital to ensure reliable monitoring and prevention of attacks in real time , focusing on operating models and preventing them.