A year after the
publication of our technical analysis of the botnet based on the Linux / Mumblehard malware, we are pleased to announce its successful dismantling. ESET, together with the
Ukrainian cyberpolice and
CyS Centrum LLC , were able to dismantle the Mumblehard botnet, stopping its spamming activity since February 29, 2016.

ESET has applied a well-known mechanism called sinkhole for all known Mumblehard components. We shared the data obtained as a result of this operation with the
CERT-Bund organization. This organization notified the CERT centers of other countries in the event of the presence of Mumblehard bots there.
')
A month after the publication of a study on Mumblehard last year, our researchers noticed the reaction of the intruders who were behind this botnet. In our report, we mentioned the process of registering domain names, which were the addresses of alternative C & C servers Mumblehard. This process was used to calculate the size of the botnet and track the spread of malware. The authors of Mumblehard responded to these actions by removing unnecessary domain addresses and IP addresses from the list of managing C & C servers. However, they left only the one that was under their direct control.

Fig. 1. Differences between the Mumblehard backdoor Perl code before and after the publication of a technical study of ESET.
The first time we saw an updated version on May 24, 2015. Her sample appeared on the VirusTotal service the very next day.
The bots that existed at that time were not updated at the same time. Our sinkhole mechanism allowed us to fix the update of the first part of the bots, on their total number of 500 machines, on the 25th of May. The rest received an update a month later, on June 26th.

Fig. 2. Statistics of the activity of bots Mumblehard after publication.
The attackers switched to using the same C & C server address for the Mumblehard backdoor without providing a backup option. Getting control over it, followed by its closure, was a sufficient condition for the violation of the botnet. For this, an initiative was taken from our side, after which we contacted the law enforcement agencies.
With the help of the Ukrainian branch of cyber police and CyS Centrum LLC, we were able to obtain information from the C & C server manager in October 2015. Forensic analysis revealed that most of our initial assumptions about the size of the botnet and its goals were correct: sending spam was the main task of the botnet. We also found several different control panels that were used to manage the botnet.
As with the other components of Mumblehard, the control panel was written in the Perl programming language.

Fig. 3. The control panel of the botnet that was used to send commands to the malicious program.

Fig. 4. The control panel displays the status of the spam job.
In our previous study, the following question remained open: what was the original vector of user compromise by the Mumblehard malware? We knew that some users were compromised by this backdoor through an unclosed vulnerability in one of the content management platforms (CMS), such as WordPress or Joomla. For this purpose, vulnerability in one of their plugins could also be used. Forensic analysis of the manager of the C & C server backdoor showed that the computers were not compromised through this server. The scripts we discovered could be run only in the environment where the PHP shell was already installed. It is possible that Mumblehard operators bought access to these computers of potential victims from another gang.

Fig. 5. A control panel for sending commands to the PHP command interpreter list.
Another interesting feature of the work of Mumblehard is the mechanism of automatically excluding the IP addresses of bots from the Spamhaus Composite Blocking List (CBL) address list. The attackers used a special script to track the contents of this list for the presence of the IP addresses of each of the spam bots. If one of these IP addresses is in this list, the script sent a request to exclude them from the CBL. Similar requests to exclude addresses from the list are protected using CAPTCHA, but the OCR mechanism (or its external counterpart) was used to bypass it.

Fig. 6. The status of the IP address of one of the spam bots after its removal from the CBL list.
All operations with the C & C server manager, which include interaction with a remote (infected) host, such as executing commands on the PHP interpreter, exclusion from the CBL list and similar to them, are performed using a proxy. To maintain proxy connections, the spam daemon Mumblehard listens on TCP port 39331. This function is used to mask the source of the query query. The botnet control panel checks the availability of all proxies among the victims of malware in 63 countries.

Fig. 7. The control panel displays the status of open proxy connections in different countries.
When Ukrainian cyber police experts closed down the Mumblehard C & C server manager on February 29, 2016, it was replaced by a similar one, which was managed by our experts (sinkhole). We collected data through this server during March 2016. Statistics showed almost 4 thousand systems running Linux, which were compromised by Mumblehard at the end of February. Recently, the number of compromised systems has been slowly declining.
The CERT-Bund organization recently began sending notifications to compromised organizations and users. We hope to see a significant reduction in the number of infected systems in the following weeks.

Fig. 8. Statistics of the new sinkhole server.
If you receive a notification that one of your systems has been compromised by Linux / Mumblehard, pay attention to our section on
compromise indicators (IoC) in the GitHub repository. There are methods for detecting a malicious program in the system and removing it from there.
As a preventive measure to prevent infection, you should regularly update web applications hosted on the system, including plugins. In addition, a complex password should be used for the administrator account.
ConclusionCooperation with law enforcement agencies and other important organizations was crucial, which led to the elimination of the botnet. Our experts are grateful to the cyber police of Ukraine (Cyber ​​Police of Ukraine), CyS Centrum LLC and CERT-Bund. We are pleased with the results of the operation and are proud of our efforts to organize security on the Internet. The Mumblehard-based botnet is hardly the most dangerous, the most common, or the most difficult of all, but nevertheless, its closure was another step in the right direction, which is shown by the fact that security researchers working with other serious organizations , contribute to reducing the level of cybercrime on the Internet.