The Standoff: how it was

Hello! After seeing on PHDays 9 a sufficient interest in what is happening at The Standoff in the ranks of the defenders, we decided to talk about how the preparation took place and the “Confrontation” itself through the eyes of the Jet CSIRT as part of the Jet Security Team.

Guo Standoff, I created

Somewhat similarly, our colleagues reported that we once again participate in The Standoff, and we naturally agreed.

Immediately it should be said that for defenders this year the format of the competition has changed somewhat. All teams received very similar office infrastructures, and this enabled the organizers to enter the rating of defenders in certain parrots. And for the Jet Security Team, this was the first “Confrontation”, where the office was protected, not the industrial infrastructure.
')
We got access to the infrastructure to prepare for the cyber battle in the last days of April. After the audit of the infrastructure, an entire carriage of flaws was assembled, these are just a few of them. Absolutely the entire infrastructure was not relevant patches. Passwords for all users could be obtained through Ntds.dit in cleartext. Moreover, some users had passwords from the TOP-500 list or passwords with an easily reversible hash. Hardening systems was at a level close to nothing or nothing at all. Some hosts in the DMZ had an interface to the server subnet.

According to the results of the audit, we developed certain protection measures, in turn, the organizers, after prior approval, allowed us to apply the policies we need and bring with us any means of protection and tools that can be deployed in a virtual environment. Because of the tight deadlines, some ideas on protection measures fell off even at the start. The basic settings and profiling of SZI were held during the May holidays (hello to everyone who threw pictures from picnics, we love you too), and some of the means of protection had to be tuned right before the start right on the site. Also, a number of services were forbidden to patch and reconfigure strongly. For example, one of such was Oracle Weblogic with CVE-2019-2725, PoC which was released in the very first days of May 2019.

Well, a list of what we brought with us:

• Firewall (provided by the organizers has been replaced);
• Antivirus solution;
• WAF;
• EDR;
• A pair of deception-decisions;
• A couple of vulnerability scanners;
• ELK-stack for additional analysis of Netflow;
• SIEM.

We should also tell about what was going to SIEM. The sources of events at our disposal were Windows logs, Sysmon, Auditd logs and, as it is not difficult to guess, events from the SIS itself. If there were no special problems with the first two, and we quickly coordinated the changes in terms of the Sysmon policy and configuration, then Auditd had to be worked up enough with the organizers for the config.

In parallel, we identified the main attack vectors, and based on this, we selected and adapted the relevant scenarios and correlation rules — a total of about 160 correlation rules. Plus, a set of mixed dashboards was assembled on critical nodes, GIS, and what required separate attention in the gaming infrastructure.

The standoff

For The Standoff, we decided to adhere to the concept of separating incidents into external and internal, since there was an accurate understanding that outside we would actively try to scan and exploit the web. Incidents related to scanning and attempts to circumvent the WAF were monitored separately, in a lower priority, this allowed us to focus on internal incidents. Dashboards on SZI were distributed between the defenders by areas of responsibility, and at least 2 people were assigned to each tool - for the possibility of rotation and rest.

Everything happened, as we expected. The standoff began at about 10 am, and as soon as the start was given, the SIEM system began to give up a bunch of incidents involving the external scan and the attempts of the attackers to exploit the web. In some cases, even the group did not save. At the same time, the organizers' checkers, who checked the status of various services in the office, earned it, which forced us to re-profile to some extent to cut off the false alarms associated with them.

In the very first hours of the game, the Hack.ERS team managed to find the standard credentials from the administrator (admin / admin) on the CMS of one of the resources and detect a potential LFI vulnerability. These attempts did not go unnoticed, our defenders carried out a rapid response, and the attackers were ultimately unable to advance further.

Until the end of the first game day, the methods did not change, WAF still beat off all attempts to upload something interesting to the company's websites, and the same “external addresses”, without ceasing, tried to scan our resources.



In total, for the entire event, there were 3,000 incidents related to scanning attempts, excluding grouping of events in incidents.



And about 2500 more incidents with attempts to bypass WAF, also without taking into account the grouping of events in the incidents.

Closer to the night, the intensity of all activities decreased - there were several reasons for this. Some defenders and attackers could not stand against the soundcheck and rehearsal of the concert, which was to be held the next day. Some attacking teams decided to take a pause and continue their attacks closer to the morning in the hope that the defenders and the monitoring would have less monitoring resources and some fatigue.

On the morning of the second day, the attackers changed tactics. On one of the company's websites, information about a part of its employees was posted. Hackers took advantage of this information and began to actively use user accounts through Exchange (statistics of attempts in the screenshot).



A little later, hesitant attempts were made to pick up the password on the VPN gateway, accounts that did not exist in our infrastructure participated in the brutte. Most likely, the attackers tried to use the accounts from the infrastructure that had already been hacked in the hope that the organizers left them the same everywhere. As a result, the whole situation with brute force led us to create a group of dashboards on trends in terms of user authentication. Plus, we increased the monitoring of incidents related to successful brute force, but, fortunately, these have not been identified.

Approximately an hour before the end of the game, trends showed single successful attempts to authenticate several users, including Exchange, operational analysis showed that the sources were directly user machines, most of the events indicated that the input was made by the organizers from the VMware console Vcenter.

At the same time, we recorded an internal scan from a node that successfully connected via VPN. After the operational analysis of the incident-related events, it became clear that the attacker managed to compromise the credentials of several users, and judging by the absence of unsuccessful authentication attempts, it is highly likely that the user data was “merged”.

The information was given to the defenders. For the entire response time on the personal machines of compromised users, the endpoint solution was transferred to a preventive protection mode in order to slow down the ability to gain a foothold in the system. Sessions of the attackers on the VPN gateway were forcibly dropped, and the attackers were thrown out of the protected perimeter. The compromised UZ had passwords quickly changed.

At the same time, the guys from the True0xA3 team came on stage and successfully used OSINT and reported about the compromise of the Behealthy office, which is under the protection of another team. The attackers managed to compromise the domain administrator. The organizers were informed about our incident and presented with proofs.

The last hour was especially hot due to the sudden OSINT, everyone was still waiting for some billets from the organizers. The monitoring team, in turn, monitored all suspicious anomalies and incidents, but after an unsuccessful attempt to penetrate the new hacking attempts did not follow. So the last minutes of playing time passed, and the successful hacking of the office protected by the Jet Security Team did not happen.

And some final statistics for the two game days:

• 1200 EPS on average and slightly less than 3000 EPS in peak;
• About 7,000 incidents;
• Over 120 million events.

Factors that helped us win

• Competent distribution of roles. Each set up and was engaged in most of what is involved in everyday projects. No one had to study the materials from the category of "firewall for dummies."
• Operational profiling of correlation rules. All false positives were processed and edits related to gaming infrastructure features were made.
• Rapid response. Due to the fact that several people were assigned to each type of SZI and systems, we didn’t have any problems with the fact that the person in charge was resting or simply went somewhere for a couple of minutes. All information from monitoring was processed as quickly as possible.
• Experience in hardening and monitoring various infrastructures.

PS Special thanks to everyone who came and asked questions, and we apologize to those who could not say something in the process - the team waited for “sent Cossacks” from the attackers and could not reveal all the details.

Dmitry Lifanov, an expert at the Jet CSIRT IS monitoring and incident response center