📜 ⬆️ ⬇️

"Confrontation" through the eyes of a defender: A story about PHDays CTF from a competitor



This year, the main hacker contest PHDays changed the format: we moved away from the usual CTF. Instead, the real battle of hackers and security men took place on the forum. This time three teams fought: “hackers”, “defenders” and SOC (security operations centers). The events at the competition ground were as close to reality as possible. The teams had at their disposal an emulation of the city, in which there is a bank, a telecom operator, an office of a large holding, an electric power company and other facilities.

Information security engineer of the telecom operator Andrei Dugin , who participated in the CityF on the defense side, described in detail his course of the competition and his impressions in his blog. With the permission of the author, we have collected all his publications in one habratopik.
')

Description


On May 17-18, 2016, at the Positive Hack Days conference in Moscow, the Capture The Flag (CTF) competition, characteristic of major events in practical information security, was held. This year, the organizers modified the contest somewhat, giving it the format of “confrontation” of the forces of good and evil in a virtual city, as announced on the PHDays conference website.

Naturally, there were a lot of people willing to try their hand at the competition. Participants were divided not into 2, but into 3 functional areas:


Since I participated in this competition as a member of the team of "defenders" of the infrastructure of the telecommunications operator, I will continue to talk about the event through the eyes of an information security engineer.

Advocates and SOC work in tandem, protecting their services. Our tandem was really very productive, as is customary in Russia :)

According to the results of the two days of “confrontation”, I will briefly say that we as partners in SOC got a team of professionals from JSOC , with whom we understood each other perfectly (not advertising for the sake of, but a statement of fact for). So tightly, as with JSOC in this competition, we did not always work even with “related” divisions in our company. It sounds pathetic, but it's true, you can't throw out the words from the song.



The team of telecom-defenders "You Shall Not Pass" from large telecom operators and Solar JSOC, the team "False Positive"

Hackers, of course, in the competition work separately from their opponents (so that there is no fight), trying to gain unauthorized access to the protected resources.

It turned out quite interesting, but the details of the competition later. In many cases I try to understand the original reason for any transformation in the direction of complication. It is logical that the conference should keep up with global trends, but then, perhaps, on a wave of impressions from participating in the competition, I would say that Positive Technologies seems to have even gone half a step ahead (deflection counted?) - for how long? (deflection compensation to maintain neutrality).

So, there are several obvious reasons, some of which are the result of others.

1. To draw the attention of the state and business to:


2. To give specialists a sense of what work is like with the constant threat of infrastructure being hacked in some approximation to reality (why some will discuss this later).

3. Make the event scale.

4. Market promotion:


Which of the above is the root cause, and what the consequence is, I think, obviously.
I do not pretend to the fullness and breadth of the analysis, because I described my viewing angle earlier.

Training


So, our national team of telecom defenders, in tandem with JSOC, defended IP-based public services specific to the telecoms operator: personal account, portal, services for sending paid messages, other VAS, etc. Of course, an integral part of our task was to ensure the security of network equipment.

Regarding the team: we had to work with competing colleagues with whom we had not known before, and I, when I only found out about it, thought that at first we, as in Avengers, would re-educate ourselves with each other, determining who cooler, and then we start the business to do. Fortunately, I was wrong, and the reason is simple: we didn’t have time for that. The unspoken principle was this: if you are cool - beautiful, well done, we believe in you, recognize your talent and do not dispute, WORK! And I just reviewed the militants.

In the “virtual city” where the events described above took place, there were 5 infrastructure facilities for which the struggle was fought:


Naturally, without normal technical support, neither the city would work, nor the means of protection. For the operational monitoring of information security events, the prevention and repulsing of hacker attacks, the organizers provided us with:


Of course, we got a remote access to our segment a bit in advance, for the commissioning and adjustment of the information security systems we need. Otherwise, they would not have time to do anything properly. It is a pity, of course, that some of this time fell on Easter and May. Confronting opposition, and potatoes require integration into the soil, and also in a short time. Each set priorities in its own way.

Special thanks to our colleague Gennady Shastin , who, in fact, gathered the whole team and competently kicked everyone with the necessary strength and intensity of the necessary organs to the point of no return. I used to know him as just a good qualified specialist, and then also the organizational skills showed up. Gena, you, in fact, alone with the pusher started a locomotive, our team is unrealistic to thank you.

There were no working places for us, only ethernet for connecting laptops and power outlets. But this is normal: we would only perplex the organizers to carry the PC back. Behind the laptop to work more habitually. Workplaces in the form of stationary PCs were available only to user statisticians who had to depict the activity of the average user: read mail, click malicious links, sit on social networks and complain that they pay very little. Interact with users was prohibited.

The network map contained only L3 information on subnets and their topological location. What network is in - it was necessary to find out independently. On the one hand - the complexity of the problem, on the other - an extra injection drive. That is, to find and neutralize servers, we used ARP tables of L3-terminating devices and scanners, deployed infrastructure elements like vsus, etc., updated, reconfigured, integrated ... All this, including network equipment, was on virtualka. At the setup meeting, Sergey Pavlov from Positive Technologies promised that there would be at least more computing power than he had on his home computer, and he did not seem to be deceived. But this equipment sweated from the load notably, we were not shy about requesting the necessary virtual machines, and the hackers were launching a carriage of scans at the same time.

On the means of protecting information: I believed at the beginning, when Gene just started kicking us that “Positives” as organizers would try to impose their products and solutions for their PR, but no, there was not only complete freedom to choose any means, but also active assistance in obtaining temporary licenses from any commercial suppliers. This is also easily explained: driving the use of their products, the organizers would limit defenders, sharply turning the confrontation into public testing and would give space for accusations of advertising and maneuver on excuses in the case of resonant hacking. Yes, I myself would try to write off some kind of tough defense in this case to the coupled decision that it is no secret. Plus, if you impose for a real confrontation - if you teach, train fairly deeply and for free, and let me give you a golden support during the competition, losing your man-hours, which was clearly not part of the organizers' plans.

And so: dear defenders and "juices", here's a sip of freedom for you, look, do not choke. Everything will be fine - well, okay, if everything is not quite good, you can always say: "But if you had our MaxPatrol scanner or MaxPatrol SIEM ...", but you can not say, depending on the situation. True, I have not heard yet that such a talk or even hinted.

In general, the preparation for the competition went very well. Of course, it’s not without some nuances like the interruption of infrastructure, when the network smokes bamboo because of the load on the hypervisor, but these are working moments that are quite normal for the first event of this scale. This year the bumps were filled, now the next one with preparation will be simpler, more predictable, and consequently, there will be time for fantasy.

Opinion on the resonance hacking hydro


Fights without rules. The duel between fighters Ork and the Goblin is organized. The fighters are beginners, but the crowd is a trump and with money. Influential fan of such fights, the official, the roof of the organizers, makes a very big bet on the fact that Goblin will win. And here Orc delivers a good blow to the body, after which the Goblin "floats." The judge, contrary to the rules, counts it as a blow to the groin, setting aside Orc with a penalty point and giving time to restore the Goblin. In the process, he quickly whispers to Ork: “You need to lie down. Dad said. " And Ork, knowing full well that even if he wins and takes the prize, in the future the career of a fighter in this organization is closed for him, he begins to think quickly, looking at the Goblin gradually coming to himself, how to substitute a “knockout” strike so that it is plausible. and convincing for the viewer.

The very next day after PHDays VI, the entire Internet was filled with news about how a schoolboy hacked into a power station and how hackers, after a successful hack, stopped a hydroelectric station and flooded the city.

I participated in the "defenders" of the telecom, so the opinion is based on an analysis of the situation and analogies from my bell tower, but without knowing the details of the SCADA hacking.

As I wrote in the first part of my series of reviews about PHDays, one of the key objectives for the organizers was to draw the attention of the state and business to the need to protect the Internet of Things and other critical infrastructure management systems connected to public networks. Although I did not work with SCADA protection myself, I see from public sources that when any device starts to be controlled via IP, and even more so via the Internet, few people pay attention to its security. As always, technologies are developed first, and then their safety. As a result, the “internet of holey things” is formed. How it can be threatened depends on which device is accessed. This can be conveyed to the public through news, and in order to attract a journalist, you need something that will be perceived as a sensation. And how much more sensational than stopping the power station and flooding the city with hackers, who are perceived, as a maximum, by boys with the posture of a monkey behind a laptop, who can steal passwords?



Image: Xakep.ru

There is a team of "defenders", who have ruined the control system. There is SOC, which connected everything to event monitoring. Details of the preparation are described in the second part of the same name. These are, as far as I know, commercial companies in the information security profile who, at this competition, probably would like to promote, and therefore will take the protection of industrial facilities seriously. And then, in spite of all the described seriousness of the protection, the hydroelectric power station is broken 2 times ... Is it really that bad with the IS of the automated process control system in Russia?

My options for what happened:

  1. Defenders poorly defended: either normal means of protection SCADA does not exist in nature, or carelessly reacted to the defense;
  2. SOC overlooked: either it was poorly integrated with monitoring or clicked;
  3. Restrictions by the organizers on the means of protection;
  4. Dad said.

With the first two everything is clear. If it’s really SOC and the defenders clicked, there’s the road. But I have a suspicion that the restrictions imposed by the organizers implied the possibility of hacking of the hydropower station by hackers. The stakes were made by the “dad” that the hydropower stations would be shut off and the city flooded.

In order not to be unsubstantiated with the statement about the restrictions on the part of the organizers, let me say as a member of the “defenders” team of the telecom: restrictions on the use of protection were, and quite strong. Use any means of protection, but from the fundamental means we were forbidden even to close the firewall from the SSH / RDP world to the protected resources, only to the network equipment itself, not to mention other unused services. Change passwords, update, patch, reconfig as much as you like, but the service should be available ... And I generally wanted to set a default deny, as is done in real life (details about the degree of proximity to real life later). But I was not allowed by the organizers.

And strategically it is right: otherwise there will be no show, hackers will whine, telling that this is all dishonest, that the organizers have given a tremendous advantage to defenders and juices. There will be no injection of the necessary information in the media; the management of industrial enterprises will dismiss protection. He sacrificed a pawn (SOC and defenders of SCADA) in this game, but received a strategic advantage for several years ahead (future orders from business and the state for security). And most importantly, the world thought about how not to let the bad guys leave people without light and with an excessive excess of water in the city.



Image: Xakep.ru

If everything developed this way, then most of all I would not want the “defenders” and the SOC who defended SCADA to be accused of breaking into the hydroelectric power station and flooding the city.

More attentive colleagues and representatives of the organizer corrected me, adding that the defenders of the hydropower plant had not really broken. After the hacking, the defenders and the SOC very accurately described the hacking process (that is, they saw everything, but did not do it, because, after all, the show was necessary) and the hackers confirmed it. It was possible to loosen the scud only after the defenders had weakened and then completely removed the defense. On the morning of the second day, it turns out, it was announced publicly from the stage. It seems that after a sleepless night I wasn’t really able to perceive what was being broadcast there.

But, nevertheless, after a few days, I see information on the Internet not about the fact that they didn’t protect the hydroelectric power station, but that it was hacked. And over time, this confused phrase will only make it clear that the hydroelectric station can be hacked. Continuing the logic - which means it needs to be protected. That is, the shadow of the good name of the “defenders” and the SOC, if it fell, will very quickly resolve. Probably, PR specialists know their business better than the data protection engineer (there wasn’t enough to be the opposite).

Here it is, the official information from the organizers:

The forum clearly showed what happens with an unsecured critical infrastructure. Information security specialists are able to provide a very high level of protection without disrupting the process - but they very rarely turn around like at PHDays. After disabling the protective equipment, the attackers penetrated the technological network of the automated control system through the corporate network, attacked the physical equipment of the system, hacked the hydropower station, discharged water, and disconnected the power lines.

Restrictions


On the official conference site there is a rather pretentious quote:

This time, instead of the usual CTF competitions, we are organizing real hostilities. Events on the site will be as close to reality as possible : a large-scale emulation of urban infrastructure will unfold at the PHDays VI CityF test site.

That's about the bold and would like to talk.

In the means of protecting information, we were practically not limited: whichever you want, put it this way, just say the requirements for the virtual machine. You need to kick a vendor for a temporary license - say, we will. Well, we were not shy.

VMware . At the deployment stage, the limitation was voiced: the solutions we chose for the competition to provide and monitor IS should be virtualized. Hardware could be taken only portable, for them the organizers were not responsible. In general, this is not such a big problem, because the virtual machine is much easier to get from the vendor, and the global trend is striving for ubiquitous virtualization, even entire networks and data centers are virtualized, making all kinds of NFV, SDN and SDDC. Some manufacturers managed to poke about the virtual team, and one even refused to provide it. I was not ready for such a serious relationship with hacker attacks.

Default permit . On the other hand, it was forbidden to execute default deny on the edge firewall. And in the real world this is done first. This angered me deeply, I cried and cursed, but continued to participate in the competition. As I wrote earlier, one of the main goals of the “confrontation” was a show that could not have happened, if I had forbidden everything that was superfluous. Hackers would have poked at the secreted interfaces, would have done nothing, and would have cheated the organizers in indulging the light side of the force. So here is an approximation to reality almost zero. They gave hackers a head start over the most tomatoes. Because of such odds and hydropower plants broke, and the city was flooded .

No ban for IP address . It was also forbidden to ban on src IP. Naturally, if this were simply forbidden by words, then I could “forget it”, especially at night, so that I could sleep more easily and in general so that I could sleep. But the cunning organizers put everyone in one IP: both hackers and their own service-checker. If we extinguished the excess, they resorted to us and said that ay-ay-ay, so it is impossible. Moreover, the services were checked at all any that were initially available on the servers, even not needed in FIG. Shaw, cho.

It also turned out to be a curiosity at the very beginning of the confrontation: SOC discovered that someone was scanning us from an IP address that is not listed on the provided scheme. Having found the junction point, I blocked it, removing the unknown subnet from the routing, despite the ban. The junction point was inside the trusted segment, so the logic is simple: there is no diagram - I consider the tab and the intrigues of the hackers to be squeezed out of our network. Naturally, at first the organizers came running in with a shout: “Everything is lost! Our all does not work! ". Then they realized that they did not reassemble the joint with our segment, and agreed with my actions.

Access to equipment . Of course, we were given admin access to both network devices and servers. After all, it is difficult to protect what you can not configure. Perhaps, but difficult. To the active network equipment level L2-access access is not given. The reason is simple: the show. When the organizers noticed despondency on hacker physiognomies and boredom on ours, they raised some more vulnerable service somewhere without warning. Well, SOC with defenders, respectively, begin: “catch him, push him!”. So that we did not interfere with unnecessary securitized settings and did not see too much, we were not given access to the switches, and we were not handed them as an object of protection.

Like, I did not forget anything. If I remember - I will add. In general, the restrictions, although there were, but not fierce. Of course, the default permit still stirs my disturbed outlook, which I occasionally continue to grumble in conversations with the organizers, but the show is a show. It was wrestling, not MMA.

Battle


It is necessary to proceed to the description of the process of opposition - the battle, so to speak, which lasted 29 hours - from 11:00 on May 17, 2016 to 16:00 on May 18, 2016.

Preliminary teams of protection and rapid response were carried out:


Naturally, the first thing that was done shortly before the start was a performance check and an inventory of resources. Immediately, some servers unknown to us were discovered, many of which, in fact, turned out to be rather leaky. As I have repeatedly written, the organizers wanted the show, and without such tricks it would not be very fun.

The network diagram (simplified, without a branch office) was approximately as follows:



The Internet was pseudo-and not real, hackers got into it from a limited segment. If the Internet were real, it would have been impossible to understand where the attack of the competitor was, and where there was an outsider hacker, and there was no guarantee that someone wouldn’t get pissed off and start DDoS.

First of all, I decided to censor the perimeter, that is, hang an ACL on the Internet side that would allow the ports of the DMZ servers detected by the scan, except for the managers (SSH / TELNET / RDP / SNMP) and databases (MySQL, Oracle), and close the rest according to the principle of default deny. Warned the organizers, and immediately received a refusal: no, all ports from the pseudo-Internet should be available. I can describe my reaction in two words “surprised” and “outraged”, which are expressed in one word together.



Having said to the organizers everything that I think about the design that was originally worn out, that we are not allowed to properly protect the infrastructure, we are forced to turn our bare ass on the Internet, that this is all hackers giveaway, etc., we accepted the new rules of the game and started working :


As I mentioned in the first part , JSOC worked with us in tandem, which integrated the infrastructure with its SIEM and monitored suspicious activity. It is worth paying tribute to the children, they not only took the infrastructure under any cap (processes, inputs, network activity) of the infrastructure, but also monitored it with extrasensory speed. Initially, until we learned each other’s IP addresses, less than a minute after the action, we heard questions from them: “who came to the xxx server from the address yyy?”, “Sshd changed on the server, what is that?” And so on ... Well, and answered them like: “ok, my own, it's me, I updated the leaky server”. If they also monitor commercial clients, then applause, I recommend.

Periodically, when the organizers felt sad and ended the story for the story on the stage, the hackers were sad, but we were bored, Misha Levin snapped his fingers and a leaky server appeared in the DMZ. , , , , , , . -, - , .



, - , , , , : ? : - ? , . .

, , , . : — , . , , , . , , ACL . — , . , . — . , .

, , . , . , , , DMZ . , . , PHP, OpenSSL SSH. , , . , , . , . SOC , , . , , , , - : , .

PHP, , 5.3.<> 5.4.<> 2013-2014 . , . , .

SSH CVE-2015-5600, CPU. . , , . OpenSSL. , , , SSH DMZ. SSH RDP, DMZ , Balabit SCB, , , . , , . , , SSH.

Despite the fact that Balabit is used as a means of controlling control sessions, it can additionally act as an integration layer when managing poorly / partially / no integrated networks, at night from May 17 to 18, the key factor of its use was the invulnerability of the proxy zorp-ssh module to CVE-2015-5600.

Since the technical solution turned out to be quite tricky and the possibilities were much greater than just protection from the mentioned vulnerability, I will write a separate post about the resulting traffic control traffic. Gene called it the “architecture of the engineer Dugin”.

At night, as expected, scans were flooded, and on the second day, the uprising of cars began, when new leaky servers began to grow like mushrooms after rain. Misha Levin waved his sleeve - and all the trump cards spilled out into the DMZ subnet.

, 18 . , , . , -, , , «» . . JSOC 5 , , IPS ( ) .

, , , SOC. SOC, , . , . . .



16:00 , , , . , .

Access was only possible through the hypervisor, and while we were restoring it with the network organizer, it was time for all participants to rest, which we began to do with great pleasure.

Custom SSH Security Method


— , ? .
— , -. , . : « ?». — , . .
— . -, , ?
— — , , .

— Positive Hack Days Balabit ? ?
— . .
— , .

These are two real conversations in preparation for PHDays. Until a certain point, I myself did not think that they would be so tightly interconnected. Naturally, after talking with the organizers, I thought that I should be surprised, I threatened. In the course of the competition, the mood vector slowly shifted from the initial desire in the direction of "concreted". But miraculously it turned out that in the end they merged. And this story will go.

As I wrote in Part 5 , and spoke on the PHDays scene, a rather interesting and non-standard solution turned out to protect the vulnerable Unix servers managed via SSH.

Let me remind you, the architecture itself looked simplistic as follows:



But, as mentioned in Part 4 and Part 5 , ACL firewall default deny . DMZ . SSH.

, DMZ, CVE-2015-5600, CPU. 2 — : « — ». , , , . SSH . , , . , , — . , . — . , , Open Source Unix, , 3 .

Balabit SCB, , , . Balabit , , SSH .

Balabit, . SSH zorp-ssh CVE-2015-5600, , MaxPatrol, . , , SSH DMZ , .

Consider the example of SSH, but in the same way you can do with RDP, and with Telnet, and with VNC, and with ICA (although a little sweat).

According to the idea implemented in the network diagram, hackers got to the servers from the DMZ via all protocols and ports, passing through the border firewall.



Since CheckPoint was used as a border firewall, not the ACL on the router, standard NAT-type firewalls could be configured to its fullest extent. As a firewall that protects the data center server - Cisco ASAv.
In order not to confuse the network terminology and the implementation of the NAT of two different manufacturers, I will describe the vendor-independent concept applicable to the equipment of any manufacturers that support this technology.

To turn SSH traffic on Balabit, you can go in the following ways:

1. Dedicated ports for servers.














Is done. The SSH session is not functionally broken, other traffic is not affected, the conditions of the game are met.

2. Dedicated address pool.




You can, of course, not hide the SRC IP for the CheckPoint address, but then it is necessary that the DMZ be built on gray addresses. On the "confrontation" DMZ built on the "white" ...










where Balabit_ip0 is the Balabit address configured on the interface as the main one, and not IP alias (Balabit_ip1 ... 4). You can, of course, configure it so that the src ip is Balabit_ip1 ... 4, but in the situation on PHDays it was superfluous.

Is done. The SSH session is not functionally broken, other traffic is not affected, the conditions of the game are met.

***


, DMZ Balabit, firewall :



, , :



, . « » . :


, — Balabit . , , , 5-10 , , …

, L3 , . , , - . MPLS L3VPN vrf lite ( ) Balabit firewall.
…

: PHDays?
: №1 – .

findings


, , . , , , . : default permit firewall DMZ. , checklist, .

, SANS Top 20 Critical Security Controls, . , . (DEF) SOC, , .

-, , SOC , , . - — , «»

NoControl nameDEFSOC
oneInventory
of Authorized and Unauthorized Devices
++
2Inventory,of Authorized and Unauthorized Software++
3Secure
Configurations for Hardware and Software on Mobile Device Laptops,
Workstations, and Servers
++
fourContinuous
Vulnerability Assessment and Remediation
++
fiveControlled
Use of Administrative Privileges
++
6Maintenance,
Monitoring, and Analysis of Audit Logs
+
7Email
and Web Browser Protections
+
eightMalware
Defenses
++
9Limitation
and Control of Network Ports, Protocols, and Services
+
tenData
Recovery Capability
elevenSecure
Configurations for Network Devices such as Firewall Routers, and Switches
+
12Boundary
Defense
+
13Data
Protection
++
14Controlled
Access Based on the Need to Know
++
15Wireless
Access Control
sixteenAccount
Monitoring and Control
+
17Security
Skills Assessment and Appropriate Training to Fill Gaps
++
18Application
Software Security
+
nineteenIncident
Response and Management
+
20Penetration
Tests and Red Team Exercises
++

, , «». , .

:

one.
2
3

:

one.
2
3

, , - — , .

, , , , :).

PS PHDays «», - « ». .

:

Source: https://habr.com/ru/post/304638/


All Articles