Clouds against the cleaver, or the Chronicle of DDoS attacks on cvk2012.org

The Highload Lab blog on Habrahabr is updated infrequently - interesting DDoS attacks, by the example of which everyone can learn something, are few and annoying readers do not feel like annoying success. The more valuable are really entertaining precedents, among which, of course, is the attack on the sites of the Central Election Committee of the Opposition Coordinating Council.

In previous releases of the logbook of the Qrator traffic filtering network (a mandatory advertising link is http://qrator.net/ ), we found that in anticipation of serious events in Runet, it is better to take care of attacks against attacks in advance. Today I will tell you how you can, even by fulfilling this requirement, earn yourself a couple of sleepless nights at the workplace admin.

Prehistory

Employees of the CEC approached the problem of resiliency of the frontend of the voting system with all academic seriousness and did not put all the eggs in one basket. The “muzzle” of the voting system was placed in the Microsoft Azure cloud , recommended for applications on the Windows platform. In parallel, the election.cvk2012.org domain was registered with Qrator almost a month before the events described and all this time was under preventive protection, which made it possible to accumulate the necessary basis for training. All this made it possible to expect that the election weekend in terms of the efficiency of the opposition’s sites would be as calm and seamless as the previous meetings.
')

October 18

The first bell rang on Thursday. By this time, a low-power attack on the voter registration site had been going on for 3 days, but at that time it had created problems for the first time. The attack scheme is a typical JS LOIC : on the supposedly stable hosting a page with the ECMAScript code is placed that updates the pseudo image with the URL of the attacked service every 200 ms. A similar scheme in April was used by Vkontakte for “warning” the antigate.com site. In this case, at first, the attacking script was located on the site hellotesak.narod.ru , while the Yandex servers did not take part in the attack — the script was executed in users' browsers. This site was later blocked.



It turned out that under some conditions, the attacker code creates serious problems for the database. Since there was no time left to optimize frontline communication with the DBMS, and the site didn’t have a performance margin to analyze the history of all incoming requests, after consulting Qrator technical support, the customer decided to closely integrate the application with the filtering system, providing it with feedback on the complexity of the database query .



Thus, the site was protected from attacks until Saturday.

The 20th of October

On Saturday evening, the attackers turned their attention to the Microsoft Azure voting site - election.democratia2.ru . The cloud itself continued to work, but the voting server located elsewhere was successfully killed by a targeted LOIC attack — this time from cvkhello.do.am . Initially, the development team tried to overcome the trouble by changing the architecture, then introduced a captcha, but at midnight the voting site was still put under the protection of Qrator ... without HTTPS - they forgot about it in a hurry. When Qrator got access to SSL, in addition, there was a lack of accumulated behavioral history of legitimate users.

The work of election.democratia2.ru was restored to one in the morning on Sunday, October 21, after learning the filtration system.

According to information from Meddy (an employee of uCoz), the site cvkhello.do.am existed for two days. Registration of the site took place from an IP address belonging to one of the largest Ukrainian cable providers.

October 21

Of course, an urgent rejection of Azure and large-scale changes in the code could not but have a negative effect on the overall performance of the site. And on Sunday, the attackers finally came to use, along with LOIC, a full-fledged botnet with IP addresses from Asian and European countries. The total number of bots that participated in the attack was a little over 130 thousand without taking into account NAT. All this together led to the need to preemptively block a number of potentially suspicious IP addresses, allowing them to the site one by one and analyzing the behavior history of each of them individually. Towards evening, when the bottlenecks in the performance of the protected site were eliminated, access to the site was granted to all legitimate users.

Dry technical details of the attack

• Attack type: combined, SYN-flood + application-level attack (botnet + LOIC)
• Peak SYN-flooding power: 150 thousand packets / s
• Peak Attack Level Attack Power: 3.8 thousand HTTP requests / s
• Number of IP addresses involved in the attack: approximately 135 thousand

Morality?

• The cloud architecture of the application is not a panacea for all ills. Even with a favorable set of circumstances, you face a multi-valued bill for the resources expended by the cloud on the full processing of parasitic traffic. A less optimistic forecast is serious problems with stability and productivity and, as a result, downtime at the most crucial moment.
• No highly loaded application can be relieved from load testing. Neither placement in the cloud, nor the reputation of the server platform and framework, nor the pre-purchased gigabit channel guarantee the absence of bottlenecks in the scripts, in the database schema, in the query processing logic. A loaded application should have a margin in performance and withstand at least a load of 115% of the calculated or standard load - only this will ensure high-quality filtering with a minimum of false positives.
• By the way, about performance. If your server often thinks of the answer for more than a second, it is very likely that you have big problems that need to be resolved before the system is put into production!
• If the site is already inaccessible from the outside, keep calm and soberly assess your capabilities. Hot patching a site right on the production might close one hole, but it will open three new ones and cut you back. Consider all the available opportunities for salvation and select the best one in terms of price and quality.
• I repeat: the sooner your application is protected, the more free time you and your system administrators will have to work, rest and fight for their ideals.