Anti-DDoS Voxility: what the pizzeria war taught us
You can not even imagine what dramas can unfold in such a seemingly peaceful industry, like pizza trading. In one of them, we had the opportunity to participate as a cloud provider: in December 2013, Pizza Empire, one of Cloud4Y's clients and the largest pizza delivery network in Moscow and the Moscow region, began an intensive expansion into new territories that other players already attended .
And pizza has a dark side
All anything, but the imperial price policy in the provision of fast food services for its competitors had a dumping character. The war began on all fronts, and soon DDoS attacks began to turn.
The Empire of Pizza has launched an active advertising campaign in a large network of cities in the Moscow region, ranging from printed products and ending with an online advertising campaign. The problems began when competitors — other pizza chains — noticed this activity and switched to an offensive.
')
On December 24, 2013, the pizzeria network war turned into high technology fields. The DDoS attack of all the Internet resources of our client, which were located with us, began.
In the first stages of the beginning of the attack on the "Empire", we removed their Internet resources from the attack by moving to other address ranges. Such a policy did not work for very long, because the attackers tracked the location of the client and the botnet was quickly reconfigured. The attack after a short period of time moved again towards the client, blocking its services. This continued until December 31, 2013, after which a massive attack of all address ranges of Cloud4Y began.
On January 1, 2014, a massive DDoS attack of the entire address range of Cloud4Y, where the client was located, began. The intensity of the attack scored the entire external peering with other operators and hit the ceiling of the bandwidth of all external communication channels.
As a matter of urgency, the flood sensors of the built-in DDoS protection technologies on the grouping of external routers were reconfigured, however, they did not bring enough benefit in this situation. The routers themselves successfully maintained external routing volumes, but problems also began with higher-level telecom operators (part of the external peers periodically went down, not coping with the switching and routing of such data streams).
In attempts to somehow unload external communication channels, we asked external operators to restrict UDP streams in order to allocate free bandwidths for the work of the services of our other Clients.
Internet services of our client, despite any attempts of service administrators, were inoperable. Servers lost control after a few seconds after launch. Firewall rules and restrictions did not help. In order to eliminate the impact of an attack on the work of other Cloud4Y clients, the virtual resources of the pizzeria network were moved to individual hypervisors. However, due to the intensity of the attack, the SLA services provided by Cloud4Y suffered enough:
More information about the incident can be found on our website (SLA-monitor is available on the main page). Despite such an intense DDoS attack, Cloud4Y cloud services were accessible to external clients.
Our client, in search of a solution to resume the operation of its Internet resources, began to duplicate its resources at the facilities of other cloud operators. After transferring the DNS to new locations, other cloud operators (we will not call them) almost immediately became inaccessible (since we also monitored the situation not only from the interests of curiosity).
Having wandered across the expanses of the runet, the Internet resources of the pizzeria network constantly moved between the data centers of Europe, where the picture with the consequences of the attack was repeated. The latest data centers, where our client switched, were in Europe (Netherlands, Germany, Czech Republic). At the time of this writing, the client’s Internet resources were located in a data center on the islands of Virginia.
I think our managers were not very upset after moving this client, because with his switch to other data centers, the intensity of the attack on Cloud4Y dropped, but the preventive “pouring shit” into the addressing area of ​​the former client’s location lasted a few more weeks.
Once under this distribution, it was decided to maximize the DDoS protection of the external Cloud4Y peering, as a result of which we had a direct dedicated channel to Voxility (the leading operator of protection against all types of attacks, including DDoS).
Having received maximum protection from DDoS as a service, during the year after the last attack, Cloud4Y had temporary workloads related to the activity of clients, as well as small loads due to possibly small and short-term DDoS attacks. But the attacks of power, with which digging in this very pizzeria, no longer happened. What to do?
In order to test Voxility's DDoS protection in action, Cloud4Y decided to attack itself, for which a separate subnet was allocated, separate virtualization tools and separate Internet resources (guinea pigs) were created, on which it was planned to conduct the toughest tests.
To carry out tests through the Internet, the performers of our order were found who could realize a DDoS attack of this level, and which we experienced at the end of 2013 - the beginning of 2014. Such performers always work anonymously and there was no direct contact with them by phone, however, to realize their plans, they had to be in constant contact with us in order to be able to stop the attack if something went wrong. Such a performer was found.
Previously, the grouping of external routers was reconfigured and the subnets attacked with short BGP prefixes were routed through Voxility. The protection technology implemented by Voxility has the following architecture:
Tests were planned in two stages:
In the process of testing, we supplemented this plan with the third stage, which will be described later. Before testing, we carried out self-testing, i.e. initiated a DDoS attack of their resources by their own means. The attacked rabbits were located in our data center in Moscow, while we initiated our attack from our Data Center in the Netherlands. We will not divulge the attack technology, but we can say that we managed to successfully overwhelm our rabbits through Voxility, and our attack remained even unnoticed by the Voxility cloud and the rabbit was attacked with technological attack. Say at once - it was not a flood attack, the purpose of which was to score all external channels. Thus, we only experienced the technological advantages of Voxility in detecting complex types of attacks. Unfortunately, he didn’t find a complex attack, guinea pigs armed with NGINX-based caching systems were successfully overwhelmed for a long time, and Voxility’s sensors didn’t detect and deactivate our invasion.
Starting a self-test, we quite strongly risked the fact that:
1. Protection of Voxility could not cope;
2. We could lose contact with the attackers and could not stop the attack (communication with the attackers was carried out exclusively by ICQ);
3. In the event of a peering crash with Voxility, we could get a real DDoS attack on all of our resources, the consequences of which could be similar to the New Year and New Year's attack of 2013-2014.
The attack with a small power (about 300 Mbit) was initially missed by Voxility and during the first 10 minutes our guinea pig produced an error 502. After 10 minutes Voxility detected our test attack and switched to the protection mode of our communication channel. Guinea pigs came to life, the work of services was restored. Voxility reported to us about the last DDoS attack and its reflection:
After 20 minutes of trial attack, we asked the attackers to raise the intensity of the attack to 15 gigabits. The reaction of our communication channel with Voxility at the first and second stage is reflected in the graph below:
Voxility successfully repelled the second wave of the attack, despite the fact that it exceeded the first trial 45 times. The intensity of the second wave of attack corresponded to the intensity of the attack, which we grabbed for the New Year holidays on the Internet resources of the Moscow Pizzeria Network.
Services on guinea pigs remained operational, Voxility completely closed some services on the attacked subnet (ICMP).
Having made sure that the protection of Voxility works, as well as the absence of problems with our external peering, we asked the attackers to show everything they can, and maybe they turned out to be quite a lot (the third wave is visible on the graph):
The intensity of the attack was raised from 15 gigabit to 50-120 gigabit, and as we can see part of the SYN traffic, Voxility did not have time to study the cloud, and some of the malicious traffic began to get into our channel, but in any case, half - remained available for the passage of useful Client traffic. On our monitoring systems, the attack looked as follows:
According to the results of the test, we concluded that there is no universal and perfect protection against DDoS and other types of attacks, however, our applied protection against VDoS "as a service" DDoS attacks works and allows us to provide customer service even in the most adverse conditions.
Thank you for your attention, I will be happy to answer your questions.
War is war, and hosting is scheduled!
And pizza has a dark side
All anything, but the imperial price policy in the provision of fast food services for its competitors had a dumping character. The war began on all fronts, and soon DDoS attacks began to turn.
Start
The Empire of Pizza has launched an active advertising campaign in a large network of cities in the Moscow region, ranging from printed products and ending with an online advertising campaign. The problems began when competitors — other pizza chains — noticed this activity and switched to an offensive.
')
First offensive
On December 24, 2013, the pizzeria network war turned into high technology fields. The DDoS attack of all the Internet resources of our client, which were located with us, began.
In the first stages of the beginning of the attack on the "Empire", we removed their Internet resources from the attack by moving to other address ranges. Such a policy did not work for very long, because the attackers tracked the location of the client and the botnet was quickly reconfigured. The attack after a short period of time moved again towards the client, blocking its services. This continued until December 31, 2013, after which a massive attack of all address ranges of Cloud4Y began.
Start a mass attack
On January 1, 2014, a massive DDoS attack of the entire address range of Cloud4Y, where the client was located, began. The intensity of the attack scored the entire external peering with other operators and hit the ceiling of the bandwidth of all external communication channels.
As a matter of urgency, the flood sensors of the built-in DDoS protection technologies on the grouping of external routers were reconfigured, however, they did not bring enough benefit in this situation. The routers themselves successfully maintained external routing volumes, but problems also began with higher-level telecom operators (part of the external peers periodically went down, not coping with the switching and routing of such data streams).
In attempts to somehow unload external communication channels, we asked external operators to restrict UDP streams in order to allocate free bandwidths for the work of the services of our other Clients.
Internet services of our client, despite any attempts of service administrators, were inoperable. Servers lost control after a few seconds after launch. Firewall rules and restrictions did not help. In order to eliminate the impact of an attack on the work of other Cloud4Y clients, the virtual resources of the pizzeria network were moved to individual hypervisors. However, due to the intensity of the attack, the SLA services provided by Cloud4Y suffered enough:
More information about the incident can be found on our website (SLA-monitor is available on the main page). Despite such an intense DDoS attack, Cloud4Y cloud services were accessible to external clients.
"Gypsum is removed, the client leaves"
Our client, in search of a solution to resume the operation of its Internet resources, began to duplicate its resources at the facilities of other cloud operators. After transferring the DNS to new locations, other cloud operators (we will not call them) almost immediately became inaccessible (since we also monitored the situation not only from the interests of curiosity).
Having wandered across the expanses of the runet, the Internet resources of the pizzeria network constantly moved between the data centers of Europe, where the picture with the consequences of the attack was repeated. The latest data centers, where our client switched, were in Europe (Netherlands, Germany, Czech Republic). At the time of this writing, the client’s Internet resources were located in a data center on the islands of Virginia.
I think our managers were not very upset after moving this client, because with his switch to other data centers, the intensity of the attack on Cloud4Y dropped, but the preventive “pouring shit” into the addressing area of ​​the former client’s location lasted a few more weeks.
Conclusion
Once under this distribution, it was decided to maximize the DDoS protection of the external Cloud4Y peering, as a result of which we had a direct dedicated channel to Voxility (the leading operator of protection against all types of attacks, including DDoS).
Having received maximum protection from DDoS as a service, during the year after the last attack, Cloud4Y had temporary workloads related to the activity of clients, as well as small loads due to possibly small and short-term DDoS attacks. But the attacks of power, with which digging in this very pizzeria, no longer happened. What to do?
Tests
In order to test Voxility's DDoS protection in action, Cloud4Y decided to attack itself, for which a separate subnet was allocated, separate virtualization tools and separate Internet resources (guinea pigs) were created, on which it was planned to conduct the toughest tests.
Training
To carry out tests through the Internet, the performers of our order were found who could realize a DDoS attack of this level, and which we experienced at the end of 2013 - the beginning of 2014. Such performers always work anonymously and there was no direct contact with them by phone, however, to realize their plans, they had to be in constant contact with us in order to be able to stop the attack if something went wrong. Such a performer was found.
Previously, the grouping of external routers was reconfigured and the subnets attacked with short BGP prefixes were routed through Voxility. The protection technology implemented by Voxility has the following architecture:
- The visitor of the protected resource first of all gets into the Voxility protection cloud, where by default all traffic is initially passed only in the “Analysis” mode.
- In case of detection of hazards, the Voxility cloud switches to protection mode and access to services is skipped only if the traffic composition meets the Analytical Purity Rules.
- If necessary, the traffic to the service is cleared of malicious components and enters the protected services for processing.
- The traffic processed by the terminal service returns to the consumer requesting it.
Tests were planned in two stages:
- Trial attack of guinea pigs with an intensity of 300 Mbit per second for 30 minutes (in order to determine the Voxility reaction).
- The main attack with an intensity of 15 Gbit. Per second for 2 hours.
In the process of testing, we supplemented this plan with the third stage, which will be described later. Before testing, we carried out self-testing, i.e. initiated a DDoS attack of their resources by their own means. The attacked rabbits were located in our data center in Moscow, while we initiated our attack from our Data Center in the Netherlands. We will not divulge the attack technology, but we can say that we managed to successfully overwhelm our rabbits through Voxility, and our attack remained even unnoticed by the Voxility cloud and the rabbit was attacked with technological attack. Say at once - it was not a flood attack, the purpose of which was to score all external channels. Thus, we only experienced the technological advantages of Voxility in detecting complex types of attacks. Unfortunately, he didn’t find a complex attack, guinea pigs armed with NGINX-based caching systems were successfully overwhelmed for a long time, and Voxility’s sensors didn’t detect and deactivate our invasion.
Testing
Starting a self-test, we quite strongly risked the fact that:
1. Protection of Voxility could not cope;
2. We could lose contact with the attackers and could not stop the attack (communication with the attackers was carried out exclusively by ICQ);
3. In the event of a peering crash with Voxility, we could get a real DDoS attack on all of our resources, the consequences of which could be similar to the New Year and New Year's attack of 2013-2014.
First stage
The attack with a small power (about 300 Mbit) was initially missed by Voxility and during the first 10 minutes our guinea pig produced an error 502. After 10 minutes Voxility detected our test attack and switched to the protection mode of our communication channel. Guinea pigs came to life, the work of services was restored. Voxility reported to us about the last DDoS attack and its reflection:
Second phase
After 20 minutes of trial attack, we asked the attackers to raise the intensity of the attack to 15 gigabits. The reaction of our communication channel with Voxility at the first and second stage is reflected in the graph below:
Voxility successfully repelled the second wave of the attack, despite the fact that it exceeded the first trial 45 times. The intensity of the second wave of attack corresponded to the intensity of the attack, which we grabbed for the New Year holidays on the Internet resources of the Moscow Pizzeria Network.
Services on guinea pigs remained operational, Voxility completely closed some services on the attacked subnet (ICMP).
Third stage
Having made sure that the protection of Voxility works, as well as the absence of problems with our external peering, we asked the attackers to show everything they can, and maybe they turned out to be quite a lot (the third wave is visible on the graph):
The intensity of the attack was raised from 15 gigabit to 50-120 gigabit, and as we can see part of the SYN traffic, Voxility did not have time to study the cloud, and some of the malicious traffic began to get into our channel, but in any case, half - remained available for the passage of useful Client traffic. On our monitoring systems, the attack looked as follows:
Total
According to the results of the test, we concluded that there is no universal and perfect protection against DDoS and other types of attacks, however, our applied protection against VDoS "as a service" DDoS attacks works and allows us to provide customer service even in the most adverse conditions.
Thank you for your attention, I will be happy to answer your questions.
War is war, and hosting is scheduled!
Source: https://habr.com/ru/post/243647/
All Articles