Experience with hacked server

I want to share my experience as a novice system administrator. It just so happened that my first pancake was a lump - it was an offer to do infrastructure in one small company. The situation is complicated. No automation, everything manually and according to the principle: work - do not touch, and if it does not work and no one noticed, then consider that it works. The previous employee left almost no documentation. Here is the access to the servers, the coffee machine is there, everything seems to be ...

Experience one: the small symptoms of a big problem

It was decided to start with the inventory. Everything is hosted by a large public operator: several virtual machines on a leased server with a preinstalled Xen hypervisor:


Experts may not agree with me, but I find KVM much more convenient than Xen for elementary virtualization of the infrastructure of small enterprises. Xen can attract those who prefer a graphical management interface. But, unfortunately, the capabilities of this graphical interface are rather limited, and the Xen command line management interface is much inferior in its simplicity to KVM / QEMU if you need to go beyond the standard manipulations of virtual machines. But it seems that this question did not particularly worry my predecessor, who managed the systems with the least effort. As it turned out, everything was installed by default, namely: the servers worked on the outdated Ubuntu 12.4 LTS, instead of the usual Debian for such purposes.
')


The mentors identified a programmer who replaced the sysadmin who had quit. Together we got to deal with the farm deployment diagram on the enterprise servers.



Here it should be said that any sysadmin, in the shower, should be a little detective. You need to be able to notice the slightest anomalies, behind which may be big problems. In this case, strange root files attracted my attention. “They probably hacked us,” a colleague shrugged, not taking my worries seriously. Simple suspicions were not enough, and we decided not to do anything special so far. We archive suspicious files, change the user's password and continue the inventory.

Experience Two: Do ​​not make the evil hackers angry

And if angry, then get ready for the consequences.

“Something with the server ...” is probably one of the most annoying messages a system administrator can get on the way to work. Especially when it comes to new work. Accelerating step, we check the situation from the phone:


Having come to work, we stumble upon a committee of anxious employees. The situation is critical. The server is unavailable, but its hypervisor is available via Xen. It becomes clear that the server is working, but the connection to the Internet has been completely lost:



So the routing problem. We climb to the operator:



At the same time we find the problem itself:



Well, now for sure: no doubt - we got hacked! And it seems that the hacker, distressed by our previous intervention, decided, once he was discovered, there was nothing to lose and used our server for a network attack. Okay, but now I’m allowed to set everything up in their own way.

First of all - firewall:



Next, change all passwords:



Save and three ssh keys. Later we will ask all users to generate new ones. We keep, for comparison: so that especially lazy users could not use the old, possibly compromised access:



And by the way, no more access with a password: only ssh keys!

Now nobody will take us by surprise. We install the surveillance of those who enter the system. To do this, add to /etc/pam.d/sshd the execution of a simple login script (loginlog):



And also spy on the keys:



Well, in the end, you can install auditd - this is a daemon that can track everything that happens on the server. Two lines in /etc/audit/audit.rules are enough for this:



That's all for today. We write to employees that in the city a new sheriff is about changes for security reasons. And tomorrow we will try to implement Nagios or even start moving to Debian. I hope the cracker will not pass. Or will it pass?