The CDN provider Cloudflare has injected the contents of its server’s memory into arbitrary webpage code.

Security experts from Google have discovered an unpleasant bug that is somewhat similar to the heartbreaked Heartbleed vulnerability in OpenSSL. It also gives anyone who wants cryptographic keys, as well as cookies, passwords, the contents of POST requests with personal data, credit cards, API keys and other contents of other people's sessions.

Here, the vulnerability is limited to only one service provider, albeit as large as Cloudflare. But in a sense, this Cloudbleed bug is worse than Heartbleed, because data leakage occurs spontaneously. These pages are routinely downloaded by crawlers, indexed by search engines, and are still stored in the archives of web pages and in the Google cache.

Cloudflare is an intermediary between the host site and visitors to the site, acting as a reverse proxy for web sites. Due to the error of the Cloudflare programmer on Nginx, from September 2016, random fragments of the server’s RAM were inserted into the content of web pages that was issued to all users .

The issuance of fragments of RAM occurred under certain circumstances. But if you went to a website that is served by a reverse proxy, you could get a fragment of someone else's network traffic directly into the browser along with the web page.
')
Probably, in order to exploit the bug, it was necessary for a long time to massively load pages from sites serviced by Cloudflare, automatically extract private traffic and process it with regular expressions in search of interesting information. These can be session keys, passwords, credit card details, etc.


Fragment of the session of a random Fitbit user that Project Zero hackers received on request to Cloudflare

Cloudflare’s clients include Uber, OK Cupid and Fitbit (about 4,287,625 domains have been affected, including 2ip.ru, w3bsit3-dns.com.ru, avito.ru, diary.ru, forbes.ru, iphones.ru, javascript .ru, prlog.ru, rghost.ru, rosbalt.ru, searchengines.ru). It is clear that in user sessions of the same Uber, OK Cupid and Fitbit, sensitive information is transmitted about user trips, credit cards, travel routes, personal information about a person from a dating site, readings of fitness trackers and much more. Not to mention the millions of other sites that sit on Cloudflare (it controls about 5% of the web, including 11% of the sites from the top 10,000). It’s like sitting in a restaurant, where, simultaneously with the menu, the waiter serves you a part of the contents of the previous client’s wallet.

Here you can not guess where you are lucky, so that the operation involves the massive downloading and processing of information. Of course, it is advisable to know under what conditions a memory leak occurs. Google specialists managed to figure this out randomly as a result of the routine work of distilling (minimizing) the page corpus in order to optimize fuzzing , which Google does.

On February 17, 2017, a Google employee and famous hacker Tavis Ormandy stumbled upon unexpected data while minimizing the corps. It was not a normal collection of garbage, distorted or incorrectly tagged data, but something in such a strange format that Tavis Ormandi spent time debugging trying to figure out the cause of the minimization error, that is, an error in his own code. “In fact, the data was so strange that some of my colleagues from Project Zero were also interested,” Tevis writes. “It soon became clear that we were looking at fragments of uninitialized memory that are interspersed with valid data. The riddle was solved, because these uninitialized data came from the program, which just put into memory the data that I indicated. But some surrounding fragments had strings and objects that seemed to come from a reverse proxy running Cloudflare, the largest provider of CDNs. ”

So it happened. A little later, the researchers managed to reproduce the bug. Information leakage occurred with a certain combination of HTML tags on the page, which caused a bug on the Cloudflare server.

Researchers dig deeper with a bug - and among several samples received encryption keys, cookies, passwords, fragments of POST-forms and data of HTTPS requests from other users. As soon as they saw this, they immediately stopped exploiting the bug and immediately reported it to Cloudflare.

Tavis Ormandy assumes that the Cloudflare bug is associated with the ScrapeShield function that runs on their servers — it parses and obfusts the HTML. The company itself Cloudflare later confirmed that the problem really was with the parser, which obfustsiruet mail , does Server Side Excludes and Automatic HTTPS Rewrites . The bug crept in about a year ago, when this parser written in Ragel was optimized and implemented as a Nginx module. Now it turned out that the pointer error and buffer overrun were present in the parser for many years, just before there was no memory leak.

/* generated code */ if ( ++p == pe ) goto _test_eof; 

This is how the parser because of the bug processed the attribute in the <script> tag.

 script_consume_attr := ((unquoted_attr_char)* :>> (space|'/'|'>')) >{ ddctx("script consume_attr"); } @{ fhold; fgoto script_tag_parse; } $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 

Ormandy says that this bug is worse than Heartbleed in the sense that the user does not need to make a targeted request to read the server’s memory fragment. Here the secret information is forcibly issued in its browser. Most people just do not understand what it is. But these pages are downloaded by crawlers, indexed by search engines. In the case of Heartbleed, there was no such data leakage - there it was necessary to send a special request to the server where OpenSSL works.

As usual, after 90 days, Google will publish information from a bug report, a way to reproduce the bug and the desired combination of HTML tags, but Cloudflare has already closed the vulnerability. Although they didn’t respond to emails and bug reports, they noticed Tavis on Twitter for an hour.


Immediately found the right people from the security department, who quickly responded. The vulnerability was closed in 7 hours, and the first fix was released in 47 minutes.

Yesterday, Cloudflare published a report with a detailed description of the incident. She immediately warned that user SSL keys did not leak, because SSL connections are always terminated on an isolated machine with Nginx, which was not affected by this bug.