Nifiga himself went for some bread, or the story of one hacking
It all started with the fact that they turned to me (as a freelancer) for help and asked to configure exim4 so that the mailing list would not get into spam. Even carefully the link was sent to a wonderful article .
Work on a couple of hours including DNS updates, but it was not there. Having logged in as root, I turned on my favorite screen out of habit with the command screen -x and saw a curious action in my favorite folder / dev / shm . The attacker did not bother to cover the screen session, or was still working in it. And here begins the quest:
The first thing I did was to see what the attacker was doing:
')
Apparently he sent spam and ran a certain ".x" file (or was it a folder?), And also checked the ssh connection. There was an archive with php script lol.php , which I unfortunately forgot to save.
The output of the last and who commands did not show anything supernatural, root sessions for the month was not, as confirmed by the owner of the server. But…
showed established connection with IP 172.190.125.14 , which I immediately nailed.
Drew attention to / usr / sbin / sshd
Next to sshd lay sshd0
Deleting the file didn’t lead to anything:
Go ahead
Reinstalling the openssh-server and openssh-client . It seems all is well, there is no threat, nothing more was found suspicious. I decided to update the system at the same time, and the old tzdata was (hello to Medvedev!). Checked /etc/apt/sources.list and /etc/apt/sources.d . All files are in order, no left lines, dates have not changed since year. And after apt-get update imposed all security updates on Debian Lenny, including the new kernel. Well. Need to reboot. I asked just in case KVM (as it turned out for good reason) and began to wait.
The next day, provided KVM. I typed "reboot" and here on you: dozens of segmentation fault. Hair starts to turn gray, hands shake. In general, I think many represent my situation. As the saying goes, “if it works, it’s NOT TROCH!”, But after detecting the penetration, it was necessary to impose updates and reboot.
In short, he took himself in hand, began to study what was happening and loaded into single user . The mount command calls segmentation fault each time it is called, even without parameters. File system readonly, nothing can be done. / etc / fstab is fine, df also works. The date command for some reason, too, segfoltitsya. I started the disk check (software raid1) fsck.ext3 / dev / md0 - everything is fine, no deviations. What is the matter? Then I start to think that I put the system, because updated tzdata package, which is connected with time. And then the DSL connection with my provider breaks ... I reboot the modem - the connection goes up, well, nice!
Server owner is indignant, because The server has already been down for several hours, and decides to write a ticket to the Infobox support. I'm on treason, continue to poking around in the system. The most sane decision seems to me to reboot the machine and boot from liveusb, so that the disk is RW, and then according to circumstances. Began debazhit mount possible at the moment ways. gdb was not installed, there was only ldd , which showed nothing serious and export LD_DEBUG = all , which also revealed nothing supernatural. Segfolt stupidly began after the initialization of all libraries. Here KVM tells me that it was turned off. Clearly, the caliper ran up. He left the laptop and began to think further ...
While I was standing and breathing fresh air, a very educated cockroach ran into my head and said, “What if the files that cause a segfolt changed?”. No sooner said than done. Waiting for what the customer will say about the ticket support. A few minutes later he sends me the answer caliper:
Fuck myself, I thought ... I tell the client that this can not be, because fsck checked the disk and found no violations in the file system. The client writes the answer to the caliper, and at this time access to the KVM is returned, where I see all the same futile attempts to call mount , hdparm , which is not installed on the system, and work with fdisk .
The latter brought nothing more than:
Here, based on the latest Disk / dev / md0 doesn’t contain a valid partition table caliper, it turned out that the problem then appears in the partition table. Indeed, as I had not guessed. After all, fdisk has never seen the partition table of a software raid. I unsubscribe all my thoughts to the client and begin to develop a cunning plan of the cockroach. I imagine what would have ended the epic caliper and how much would it take, if the client agreed to their help. And the amount is not difficult to calculate.
I look at the date of the change / bin / mount - the time of the last server load. I reboot, again I check the date - the time of the last server load. Strange. It means that when loading it modifies this file and something needs to be done with this “something”.
/ tmp - in readonly. To upload a file to the server, you need a file system with write access. Remembering / dev / shm . I bring up the network interface, assign an IP, and download the deb package for lenny. Unpacking, running - voila! Works! Remount the file system, now it is RW. It's gone!
I check the files in / bin / and see the following picture:
And the date of change of files changes every 3 minutes and 10 seconds. I start browsing crontabs, I find nothing. To catch lsof 'om what process changes files fails. I deduce ps auxww and see that a certain process is hanging cat / sys / class / net / lo / operstate
I download the package with the kill utility, rename the file / bin / cat to / bin / cat_, and nail down the process. Files cease to be modified. Victory. It now remains to replace all modified files with original ones. I download the necessary packages and install via dpkg -i * deb , after checking the creation date of the dpkg itself. After all the changes made, cross my fingers, enter the reboot and watch the KVM window. Download is successful, the site is working. Next, I scan the infected files I copied using clamav and discover Linux.RST.B-1 FOUND . Who said there is no virus under Linux? By the way, the 2001 virus ...
Scanning sshd and ssh does not lead to anything. Apparently these are just modified ssh and sshd . The first one most likely sends the username and password when successfully connected to the server, the second one most likely lets everyone with a certain password to the server. Now there is no power to dig these files, but anyone who wishes can download and dig them: zalil.ru/32063611
PS If there is something wrong in the teams, then I apologize, many of them wrote as a keepsake. Customize exim4 also no longer a desire. I haven't asked for money yet. And for what? The main task is not fulfilled =)
PPS Hello Infobox!
Work on a couple of hours including DNS updates, but it was not there. Having logged in as root, I turned on my favorite screen out of habit with the command screen -x and saw a curious action in my favorite folder / dev / shm . The attacker did not bother to cover the screen session, or was still working in it. And here begins the quest:
The first thing I did was to see what the attacker was doing:
')
wget http: // ravenul.zzl.org / it / noi / up / 8.txt
mv 8.txt list.txt
php lol.php
php lol.php
netstat -an | grep : 22
w
rm -rf list.txt
w
rm -rf .x
netstat -an | grep : 22
Apparently he sent spam and ran a certain ".x" file (or was it a folder?), And also checked the ssh connection. There was an archive with php script lol.php , which I unfortunately forgot to save.
The output of the last and who commands did not show anything supernatural, root sessions for the month was not, as confirmed by the owner of the server. But…
$ lsof -ni | grep ssh
showed established connection with IP 172.190.125.14 , which I immediately nailed.
Drew attention to / usr / sbin / sshd
$ ls -la / usr / sbin / sshd
-rwxr-xr-x 1 root root 320724 Oct 11 23 : 29 / usr / sbin / sshd
Next to sshd lay sshd0
$ ls -la / usr / sbin / sshd0
-rwxr-xr-x 1 root root 757356 Jul 31 2010 / usr / sbin / sshd0
Deleting the file didn’t lead to anything:
$ rm -f / usr / sbin / sshd
rm : cannot remove `/ usr / sbin / sshd ': Operation not permitted
Go ahead
$ lsattr / usr / sbin / sshd
-u - ia ------------- / usr / sbin / sshd
$ chattr -aui / usr / sbin / sshd
$ rm / usr / sbin / sshd
$ lsattr / usr / bin / * | grep -v - '-------------------'
-u - ia ------------- / usr / bin / ssh
$ chattr -aui / usr / bin / ssh
$ rm / usr / bin / ssh
Reinstalling the openssh-server and openssh-client . It seems all is well, there is no threat, nothing more was found suspicious. I decided to update the system at the same time, and the old tzdata was (hello to Medvedev!). Checked /etc/apt/sources.list and /etc/apt/sources.d . All files are in order, no left lines, dates have not changed since year. And after apt-get update imposed all security updates on Debian Lenny, including the new kernel. Well. Need to reboot. I asked just in case KVM (as it turned out for good reason) and began to wait.
The next day, provided KVM. I typed "reboot" and here on you: dozens of segmentation fault. Hair starts to turn gray, hands shake. In general, I think many represent my situation. As the saying goes, “if it works, it’s NOT TROCH!”, But after detecting the penetration, it was necessary to impose updates and reboot.
In short, he took himself in hand, began to study what was happening and loaded into single user . The mount command calls segmentation fault each time it is called, even without parameters. File system readonly, nothing can be done. / etc / fstab is fine, df also works. The date command for some reason, too, segfoltitsya. I started the disk check (software raid1) fsck.ext3 / dev / md0 - everything is fine, no deviations. What is the matter? Then I start to think that I put the system, because updated tzdata package, which is connected with time. And then the DSL connection with my provider breaks ... I reboot the modem - the connection goes up, well, nice!
Server owner is indignant, because The server has already been down for several hours, and decides to write a ticket to the Infobox support. I'm on treason, continue to poking around in the system. The most sane decision seems to me to reboot the machine and boot from liveusb, so that the disk is RW, and then according to circumstances. Began debazhit mount possible at the moment ways. gdb was not installed, there was only ldd , which showed nothing serious and export LD_DEBUG = all , which also revealed nothing supernatural. Segfolt stupidly began after the initialization of all libraries. Here KVM tells me that it was turned off. Clearly, the caliper ran up. He left the laptop and began to think further ...
While I was standing and breathing fresh air, a very educated cockroach ran into my head and said, “What if the files that cause a segfolt changed?”. No sooner said than done. Waiting for what the customer will say about the ticket support. A few minutes later he sends me the answer caliper:
The partition table is damaged, it is not possible to restore it by express methods.
If you want, we can bring in our system administrators (the cost of work is 870 rubles per hour) for restoration.
Or you can do it yourself. In this case, we recommend using Gpart (http://packages.debian.org/ru/sid/gpart)
Fuck myself, I thought ... I tell the client that this can not be, because fsck checked the disk and found no violations in the file system. The client writes the answer to the caliper, and at this time access to the KVM is returned, where I see all the same futile attempts to call mount , hdparm , which is not installed on the system, and work with fdisk .
The latter brought nothing more than:
$ fdisk -l
Disk / dev / sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors / track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f0571
Device Boot Start End Blocks Id System
/ dev / sda1 1 18480 148440568 + fd Linux raid autodetect
/ dev / sda2 18481 19457 7847752 + fd Linux raid autodetect
Disk / dev / sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors / track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/ dev / sdb1 * 1 18480 148440568 + fd Linux raid autodetect
/ dev / sdb2 18481 19457 7847752 + fd Linux raid autodetect
Disk / dev / md0: 152.0 GB, 152003018752 bytes
2 heads, 4 sectors / track, 37110112 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk / dev / md0 does n't contain a valid partition table
Disk / dev / md1: 8036 MB, 8036024320 bytes
2 heads, 4 sectors / track, 1961920 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk / dev / md1 does n't contain a valid partition table
Here, based on the latest Disk / dev / md0 doesn’t contain a valid partition table caliper, it turned out that the problem then appears in the partition table. Indeed, as I had not guessed. After all, fdisk has never seen the partition table of a software raid. I unsubscribe all my thoughts to the client and begin to develop a cunning plan of the cockroach. I imagine what would have ended the epic caliper and how much would it take, if the client agreed to their help. And the amount is not difficult to calculate.
I look at the date of the change / bin / mount - the time of the last server load. I reboot, again I check the date - the time of the last server load. Strange. It means that when loading it modifies this file and something needs to be done with this “something”.
/ tmp - in readonly. To upload a file to the server, you need a file system with write access. Remembering / dev / shm . I bring up the network interface, assign an IP, and download the deb package for lenny. Unpacking, running - voila! Works! Remount the file system, now it is RW. It's gone!
I check the files in / bin / and see the following picture:
$ ls -latr / bin
-rwxr-xr-x 1 root root 96408 Nov 15 18 : 11 vdir
-rwxr-xr-x 1 root root 30896 Nov 15 18 : 11 pwd
-rwxr-xr-x 1 root root 30712 Nov 15 18 : 11 ping6
-rwxr-xr-x 1 root root 24252 Nov 15 18 : 11 nc.traditional
-rwxr-xr-x 1 root root 8612 Nov 15 18 : 11 mountpoint
-rwxr-xr-x 1 root root 68208 Nov 15 18 : 11 mount
-rwxr-xr-x 1 root root 32244 Nov 15 18 : 11 mknod
-rwxr-xr-x 1 root root 39144 Nov 15 18 : 11 loadkeys
-rwxr-xr-x 1 root root 17244 Nov 15 18 : 11 kill
-rwxr-xr-x 1 root root 9764 Nov 15 18 : 11 fgconsole
-rwxr-xr-x 1 root root 26216 Nov 15 18 : 11 false
-rwxr-xr-x 1 root root 8524 Nov 15 18 : 11 dmesg
-rwxr-xr-x 1 root root 96408 Nov 15 18 : 11 dir
-rwxr-xr-x 1 root root 51988 Nov 15 18 : 11 dd
-rwxr-xr-x 1 root root 59148 Nov 15 18 : 11 date
-rwxr-xr-x 1 root root 49440 Nov 15 18 : 11 chgrp
-rwxr-xr-x 1 root root 30956 Nov 15 18 : 11 cat
-rwxr-xr-x 1 root root 12252 Nov 15 18 : 11 bzip2recover
And the date of change of files changes every 3 minutes and 10 seconds. I start browsing crontabs, I find nothing. To catch lsof 'om what process changes files fails. I deduce ps auxww and see that a certain process is hanging cat / sys / class / net / lo / operstate
I download the package with the kill utility, rename the file / bin / cat to / bin / cat_, and nail down the process. Files cease to be modified. Victory. It now remains to replace all modified files with original ones. I download the necessary packages and install via dpkg -i * deb , after checking the creation date of the dpkg itself. After all the changes made, cross my fingers, enter the reboot and watch the KVM window. Download is successful, the site is working. Next, I scan the infected files I copied using clamav and discover Linux.RST.B-1 FOUND . Who said there is no virus under Linux? By the way, the 2001 virus ...
Scanning sshd and ssh does not lead to anything. Apparently these are just modified ssh and sshd . The first one most likely sends the username and password when successfully connected to the server, the second one most likely lets everyone with a certain password to the server. Now there is no power to dig these files, but anyone who wishes can download and dig them: zalil.ru/32063611
PS If there is something wrong in the teams, then I apologize, many of them wrote as a keepsake. Customize exim4 also no longer a desire. I haven't asked for money yet. And for what? The main task is not fulfilled =)
PPS Hello Infobox!
Source: https://habr.com/ru/post/132668/
All Articles