Something is wrong with the IDS signature

The names Snort and Suricata IDS are familiar to anyone working in the field of network security. WAF and IDS systems are those two classes of protection systems that analyze network traffic, parse top-level protocols and signal malicious or unwanted network activity. If the first system helps web servers detect and avoid attacks that are specific only to them, then the second, IDS, is able to detect attacks in all network traffic.



Many companies set up IDS to control traffic inside the corporate network. Thanks to the DPI mechanism, they collect traffic streams, look inside the packets from IP to HTTP and DCERPC, and also detect both the exploitation of vulnerabilities and the network activity of malicious programs.

')

The heart of both systems is signature sets to detect known attacks, developed by network security experts and companies around the world. We, the @attackdetection team, are also developing signatures for detecting network attacks and malicious activity. Further in the article we will discuss the new approach that we discovered, which allows disrupting the operation of IDS Suricata systems and hiding such activity.

How IDS works

Before diving into the details of this IDS bypass technique and the stage for which the technique found is applied, it is necessary to refresh the idea of ​​the general principle of IDS operation.







First of all, incoming traffic is split into TCP, UDP or other transport streams, after which the parsers mark them and break them into high-level protocols and their fields - normalizing, if necessary. The resulting decoded, decompressed, and normalized protocol fields are then checked by signature sets, which reveal whether network attacks have attempted network attacks or packets inherent in malicious activity.



Signature sets, by the way, are the product of many individual researchers and companies. Among the vendors, you will find names such as Cisco Talos, Emerging Threats, and the open ruleset now has more than 20,000 active signatures.

Common IDS Workarounds

The imperfection of IDS and errors in their software make it possible to find conditions under which they are not able to detect an attack in network traffic. Among the rather well-known techniques of bypassing the stream analysis stage, you can list the following:

• Non-standard packet fragmentation at the IP, TCP, or, for example, DCERPC levels, which IDS is sometimes unable to cope with.
• Packages with borderline or incorrect TTL or MTU values ​​may also be processed incorrectly by IDS.
• The ambiguity of perception of overlapping TCP fragments (TCP SYN numbers) can be interpreted by IDS differently than on the server or client to which this TCP traffic was intended.
• A bogus TCP FIN packet, for example, with an incorrect checksum (also called TCP un-sync), can be interpreted as the end of a session instead of ignoring.
• A different TCP session timeout between the IDS and the client can also serve as a tool for hiding attacks.

As for the protocol parsing stage and field normalization, many WAF bypass techniques can be borrowed for IDS. Their number is much larger, so we give only some of them:

• HTTP double encoding.
• Gzip-compression of the HTTP packet without the corresponding Content-Encoding header can still remain un-compressed during the normalization stage, such a technique can sometimes be found in the traffic of malicious programs.
• The use of rare encodings, such as Quoted-Printable for POP3 / IMAP, can also make some signatures powerless.

Do not forget about the bugs that are specific to each vendor IDS or third-party libraries in their composition, which can be found on a public bugtracker.

One such specific bug that allowed you to disable signature checks under certain conditions was found by the @attackdetection team in Suricata IDS, and this error could be exploited to hide such attacks as, for example, BadTunnel.



During this attack, the vulnerable client opens an HTML-page generated by the attacker, and thus sets up a UDP tunnel through the network perimeter to the attacker's server for ports 137 on both sides. After the tunnel is established, the attacker is able to spoof names within the network of the vulnerable client by sending fake responses to NBNS requests. Despite the fact that the attacker’s server took three packages, it was enough for him to answer only one of them to establish a tunnel.



The error found was that if the response to the first UDP packet from the client was an ICMP packet, for example ICMP Destination Unreachable, then, due to an inaccurate algorithm, this stream was now checked by signatures only for the ICMP protocol. Any further attacks, including the spoofing of names, went unnoticed by IDS, as they were carried out on top of the UDP tunnel. Despite the lack of a CVE number for this vulnerability, it led to the circumvention of the IDS security features.







The bypass techniques mentioned above have been known for a long time and are eliminated in modern and long-developing IDS, and specific bugs and vulnerabilities work only for unpatched versions.



Since our team is working on network security and network attacks and directly developing and testing network signatures, we could not help but notice the workarounds associated with the signatures themselves and their imperfections.

Avoid signatures

Wait, how can signatures in general be a problem?



Researchers study emerging threats and form their own understanding of how a particular attack can be detected at the network level due to operating features or other network artifacts, and then translate the resulting presentation into one or many signatures in a language that IDS understands. Due to the limited capabilities of the system or the error of the researcher, uncovered ways of exploiting vulnerabilities remain.



If the protocol and message format among the same family of malicious programs and their generation remains unchanged and the signatures work fine for them, then when exploiting vulnerabilities, the higher the protocol’s complexity and variability, the easier it is for an attacker to change the exploit without losing functionality - and bypass signatures.

Although for the most dangerous and high-profile vulnerabilities you will find a whole host of quality signatures from different vendors, some other signatures can be bypassed with simple techniques. I will cite as an example a very common error of signatures for the HTTP protocol: sometimes it is enough just to change the order of the HTTP GET arguments to bypass the signature check.





And you will be right if you think that the signatures contain checks of substrings with a fixed order of arguments, for example “? Action = checkPort” or “action = checkPort & port =”. You only need to carefully examine the signature and check if it has the same hardcode.



Other protocols and formats that are no less complicated for checking are, for example, DNS, HTML, or DCERPC, the variability of which is extremely high. Therefore, to cover the signatures of all variations of attacks and develop not only high-quality but also fast signatures, the developer needs to possess a wide range of skills and solid knowledge of network protocols.



The imperfection of IDS signatures has been talked about for a long time, and you can find the opinions of other authors in their reports: 1 , 2 , 3 .

How much does the signature weigh

As already mentioned, the speed of the signature lies in the area of ​​responsibility of the developer, and of course, more signatures require more computing resources for checks. The golden mean rule advises adding one CPU for each thousand signatures or each half gigabit of network traffic in the case of Suricata IDS.





There is a dependence on the number of signatures and on the volume of network traffic. Although this formula looks complete, it does not take into account the fact that signatures can be fast or slow, and traffic can be the most diverse. So what happens if a slow signature hits bad traffic?



Suricata IDS can print signature performance data to the log. The data about the slowest signatures gets there, forming a top with an indication of their execution time, expressed in ticks - CPU time and the number of their checks.

At the top of the magazine are the slowest signatures.







Selected signatures are those that are called slow. The top is constantly updated and on different traffic profiles it will surely consist of other signatures. This is because the signatures as a whole consist of a subset of simple checks, such as a search for a substring or a regular expression, arranged in a specific order. When checking a network packet or stream, the signature checks all its contents for all valid combinations. Thus, the verification tree for the same signature may be more branched or less, and consequently, the execution time will vary depending on the traffic being analyzed. The task of the developer is, among other things, the optimization of the signature to work on any possible traffic.



What happens if the IDS power is chosen incorrectly and it cannot cope with checking all network traffic? As a rule, if the load on the CPU cores is on average more than 80%, then the IDS is already starting to skip checking some packets. The higher the load on the cores, the more unchecked places appear in the network traffic and the higher the chance that malicious activity will go unnoticed.



What if you try to enhance this effect when the signature checks the network packets for too long? Such a scheme of operation should remove IDS from the game, forcing packets to pass and attacks. Let's start with the fact that we already have top hot signatures on live traffic, and try to enhance the effect on them.

We exploit

One such signature identifies in traffic an attempt to exploit the CVE-2013-0156 RoR YAML vulnerability code.







All HTTP traffic towards corporate web servers is checked for the presence of three lines in a strict sequence - “type”, “yaml” and “! Ruby” - and checked by a regular expression.



Before we start generating “bad” traffic, I will give some hypotheses that will help us in this:

• Finding a match of a substring is easier than proving that there is no such match.
• Regular expression checking for IDS Suricata is slower than searching for a substring.

This means that if we want long sign tests from the signature, then these checks should be unsuccessful and use regular expressions.



In order to get to the regular expression test, three substrings must be present in the package one after another.



We try to connect them in this order and run IDS for verification. To construct files with HTTP-traffic in the Pcap format from the text, I used the Cisco Talos file2pcap tool :







Another keyword_perf.log helps us see that the chain of checks has successfully reached (content matches - 3) before the regular expression (PCRE) and failed (PCRE matches - 0). If further we want to benefit from expensive PCRE checks, then we need to completely disassemble it and pick up effective traffic.







Although the task of developing a regular expression backward is easily performed manually, it is poorly automated because of such constructions as, for example, backreferences or named capture groups: I did not find any ways to automatically select a string to successfully complete a regular expression.



The minimum required string for such an expression was the following construction. To test the hypothesis that an unsuccessful search is more expensive than a successful one, we cut off the rightmost character from this line and run the regular list again.

<a type="yaml" !ruby : 32 steps, match <a type="yaml" !rub : 57 steps, no match 

It turns out that the same principle applies to regular expressions: an unsuccessful test took more steps than its successful counterpart. In this case, the difference was more than 50%. You can see for yourself.



Another interesting fact opened up with further study of this regular expression. If we repeatedly duplicate the minimum required line without the last character, it is reasonable to expect an increase in the number of steps to complete the test, but the dependence of such growth is completely explosive:

 2 x (<a type="yaml" !rub) : 209 steps 10 x (<a type="yaml" !rub) : 9885 steps 100 x (<a type="yaml" !rub) : timeout 

The scan time of several dozen such lines is already about a second, and increasing their number, we rest on the timeout error. This effect in regular expressions is called Catastrophic Backtracking and many articles are devoted to it, including on Habré . Such errors can be found in popular products to this day; for example, recently found one in the Apache Struts framework.



Run with the found strings back to Suricata IDS and check them:

  Keyword Ticks Checks Matches -------- -------- ------- -------- content 19135 4 3 pcre 1180797 1 0 

However, instead of fanfare and Catastrophic Backtracking, we get a barely perceptible load for IDS - just 1 million ticks. This is the story of how, after debugging, studying the source code of Suricata IDS and used inside the libpcre library, I came across PCRE-limits, namely:

• MATCH_LIMIT DEFAULT = 3500
• MATCH_LIMIT_RECURSION_DEFAULT = 1500

These limits also restrict regular expressions from falling into catastrophes in many regular expression libraries. The same limits can be found in WAF, where regular expressions checks prevail. These limits, of course, can be changed in the IDS configuration, but are distributed by default and are not recommended for change.



Operating only regular expression will not help us achieve the desired result. But what if to check with IDS a network package with such content?





In this case, we get the following values ​​in the log:

 Keyword Avg. Ticks Checks Matches -------- ---------- ------- -------- content 3338 7 6 pcre 12052 3 0 

The number of checks was 4, and it became 7 only because of the duplication of the original line. Although the mechanism remained unclear, we should expect an avalanche-like increase in the number of checks if we duplicate the lines yet. In the end, I was able to achieve the following values:

 Keyword Avg. Ticks Checks Matches -------- ---------- ------- -------- content 1508 1507 pcre 1492 0 

The total number of checks for substrings and regular expressions does not exceed 3000, no matter what content is checked by the signature. Obviously, the IDS itself also has an internal limiter, which this time is called inspection-recursion-limit and is equal to 3000 by default. With all the number of limits in PCRE, IDS and restrictions on the one-time amount of content that gets checked, if modified content and use avalanche-like regular expression tests, the result is what you need:

 Keyword Avg. Ticks Checks Matches -------- ---------- ------- -------- content 3626 1508 1507 pcre 1587144 1492 0 

Despite the fact that the complexity of one test of a regular expression has not changed, the number of such checks has increased significantly and reached one and a half thousand. Multiplying the number of checks for the average number of cycles spent on each check, we get the cherished 3 billion ticks.

 Num Rule Avg Ticks -------- ------------ ----------- 1 2016204 3302218139 

And this is more than a thousandfold gain! Operation requires only the curl utility to compose a minimal HTTP POST request. It will look like this:







The minimum set of HTTP fields and HTTP body with a repeating pattern.

Such content cannot be infinitely large and force IDS to spend a huge amount of resources on checking it, since despite the fact that TCP segments are connected into a single stream stream, the stream and the collected HTTP packets are not checked completely, whatever they were not big. Instead, they are checked in small pieces about 3-4 kilobytes in size. This size of the segments to check, as well as the depth of the checks, are set in the config file (exactly like everything in IDS). The segment size is slightly “shaking” from launch to launch to avoid attacks on the boundary of fragmentation of such segments — when an attacker, knowing the default size of segments, can split up network packets so that the attack is divided into two adjacent segments and cannot be detected by a signature.



So, in our hands turned out to be powerful weapons, loading IDS onto more than 3,000,000,000 CPU ticks per application. What does this even mean?

In fact, the resulting figure is about 1 second of work of the average CPU. The simple relationship is that by sending one HTTP request of 3 KB in size, we load IDS for a full second. The more cores in IDS, the more data streams it can process simultaneously.







Do not forget that IDS is not worth doing and, as a rule, it takes up some of its resources by checking background network traffic, thereby reducing the threshold for this attack.



Taking measurements on a working IDS configuration with 8/40 cores of Intel Xeon E5-2650 v3 2.30 Ghz CPUs without background traffic, the threshold value after which all 8 CPU cores are 100% loaded became only 250 kilobits per second. And this is for a system designed to process many gigabits of network traffic, that is, thousands of times more.



To exploit this particular signature, an attacker needs only to send about 10 HTTP requests per second to the protected web server in order to gradually fill the queue of network packets from IDS. After the buffer has been exhausted, the packets will start passing by the IDS and from this very moment the attacker can use any tools or carry out arbitrary attacks and go unnoticed by the detection systems. A constant level of malicious traffic will disable IDS until this traffic stops bombarding the internal network, and for short-term attacks the attacker can send a short spike from such packets and also achieve blindness in the detection system, but for a short period.



The operation of slow signatures cannot be detected by existing mechanisms: even though IDS has a profiling code, it cannot distinguish just a slow signature from a catastrophically slow one and automatically signal about it. It is worth noting that the signature triggering alarm also does not occur - due to the lack of suitable content.



Remember that unexplained increase in the number of checks? The IDS error did occur and led to an increase in the number of redundant checks. The vulnerability was named CVE-2017-15377 and is currently eliminated in the Suricata IDS 3.2 and 4.0 branches.



The approach described above works well for one specific instance of the signature. It is distributed as part of an open set of signatures and, as a rule, is enabled by default, but on top of the top of the hottest signatures, new ones constantly pop up, while others are still waiting for their traffic. The signature description language for IDS Snort and Suricata offers the developer a variety of handy tools, such as base64 decoding, content jumping and mathematical operations. Other combinations of checks can also cause an explosive increase in resources consumed for testing. Careful monitoring of performance data can be a starting point for exploitation. After the problem CVE-2017-15377 was resolved, we again launched Suricata IDS to check our network traffic and saw the exact same picture: the top of the hottest signatures at the top of the magazine, but with different numbers. This suggests that such signatures, as well as approaches to their operation, are numerous.



Not only IDS, but also antiviruses, WAF and many other systems are based on signature search. Therefore, a similar approach can be applied to find weaknesses in their performance. He is able to quietly stop detecting malicious activity detection systems. Network activity associated with it cannot be detected by protective equipment and anomaly detectors. For the sake of experiment, turn on the profiling setting in your detection system — and watch the top of the performance log.







Posted by : Kirill Shipulin from @attackdetection, Twitter | Telegram