How to understand that your site has been hacked?

Currently, hacking sites are becoming a massive, staged process. Sites can be hacked for the sake of using their capacities (currently popular mining, participation in a botnet, etc.). Also, attackers may be interested in stealing the contents of your resource or promoting their own content (advertising, phishing). In any case, the attackers will try to squeeze the most out of a successful hack, that is, go unnoticed as long as possible, extracting as much benefit from the hacked site.



There are obvious signs of hacking: a lot of ads flash on the pages that you didn’t post, the pages look different, you can’t log into the administrator’s office, Google and Yandex report that your site is malicious and IP appeared in spam databases.

But it is often impossible to immediately find out that the site is hacked, it is impossible - there are no obvious deviations from normal work. So what signs might indicate that your site has been hacked?
')

So where to look?

First of all, pay attention to changes in server performance and traffic to your site. If the pages load slower and the traffic is different from what you usually had, it’s time to bother.

We adjust logs

A very important point in the detection of hacking are server logs. Therefore, you should make sure that logging is configured correctly. Then, in which case, you will have all the necessary information about the events that occurred.

Consider the possible setting of logs. In the examples below, Linux is used as the OS, and nginx is used as the server.

Configuring logrotate for nginx is not particularly difficult. It is enough to create the file /etc/logrotate.d/nginx, containing the following code:

/opt/nginx/logs/*.log { daily #    missingok #      rotate 30 #  30  compress #     delaycompress #       notifempty #     create 640 user users # create <><><>. #     log-. #    ,       ,     log- postrotate #  nginx [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid` endscript } 

If possible, it is better to add automatic copying of logs to the corporate cloud or to another server, so that an attacker, if he decides to clean the logs, would not deprive you of them forever.

We look inside

So you have logs. Now it's worth checking server access logs for suspicious activity. Have many requests come from the same IP? Many requests from different IP, but the requests themselves and the intervals between them are similar? All this is another additional reason to check whether someone has already intercepted access to the site.

Similar checking can be done through parsing access logs with awk. Here are a few awk commands to help analyze access logs for nginx:

• IP addresses sorted by the number of requests:

awk '{print $1}' access.log | sort | uniq -c | sort -rn

• 10 last unique queries sorted by quantity:

awk '{print $7}' access.log | sort | uniq -c | sort -rn | tail -n 10

• Unique IP addresses sorted by the number of requests to the / administrator / page (you can substitute any one of your interest here - for example, one of those that you received using the previous command):

awk '($7 ~ /administrator/)' access.log | awk '{print $1}' | sort | uniq -c | sort -rn

• IP addresses and pages to which access was requested, sorted by the number of attempts

awk '{print $1, $7}' access.log | sort | uniq -c | sort –r

• To monitor abnormal activity, tracking of suspicious user agents can also help. For example, it is simply written “Google”, a very old version of “Internet Explorer” is indicated, or the browser is listed as a short “Mozilla”, instead of a full line indicating the version and platform. The command below will display all unique user agents sorted by the number of hits with their use:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn

We follow the "tracks"

Also, if you notice that the logs have increased dramatically in volume or in the access log many times refer to pages that you did not have - another reason to be wary. Such a change in traffic may be due to various reasons. One of them may be due to the fact that your site has become slower to load (which will be discussed later), but not all cases are so trivial. Today, there is malware that will simply redirect users to sites that an attacker needs, and it may not affect authorized users, which will help to hide the fact of hacking longer.

The most common malware often leaves behind other “traces”. New users appear in the admin panel, which no one has added there (most often with strange names), scripts and files appear on the server, which are taken from where it is not clear, and the mailbox may contain suspicious emails. It is also worth checking whether the planned tasks are new and unknown to you.

We look after our property

And to make it easy to track if anyone has changed your files, you should add a simple script to cron to check it:

 #!/bin/bash find ./ -iname '*.php' -cmin -1 > ./files.txt [ -s ./files.txt ] && echo "Subject: static files changed!" | sendmail -v user@mail.com < ./files.txt 

./ - current folder (you must specify the desired one)
'* .php' - any file with the .php extension (if you have Python scripts, then put the .py extension, and so on)
-1 - time interval indication. It should be set depending on how often you want to check if someone has tried to change your code. In this example, set to one minute.

If any files have been changed, the names of these files will be written to the files.txt file and sent to the mail. Instead of the sendmail utility, you can use any that you like more (for example, Postfix).

Controlling "gluttony"

If you notice that the site began to load more slowly, although you did not add anything, this is also a reason to check what the resources are going for. It will help a lot, of course, if you have history / logs regarding performance. This way you can see if there was a sudden increase in the load on the site, although you didn’t add any updates or new elements. If the performance is higher than you expect, then it is worth considering whether you have been hacked.

For monitoring performance, it is very convenient to use the sysstat utility.

Configuring persistent logging is done by adding the following lines to the /etc/cron.d/sysstat file:

 */15 * * * * root /usr/lib/sysstat/sa1 1 1 55 23 * * * root /usr/local/lib/sa/sa2 -A 

The first line indicates that once every 15 minutes system data will be collected. They will be stored in / var / log / sysstat / saDD, where DD is the current date. The second line, that at 23:55 will be created a full report for the past day. It will lie in / var / log / sysstat / sarDD
You can view the data for the CPU with the following command:

 sar -P ALL -f /var/log/sysstat/saDD 

Memory usage data:

 sar -r -f /var/log/sysstat/saDD 

A complete list of keys for withdrawal can be found on the official website .

Did I link to you?

It is also worth checking out all outgoing links from your site (for example, using this service or other similar ones) - have new ones appeared? Those that you did not add? In addition, it would be good to check what is displayed in the search engines regarding your site and for what phrases visitors get to you. Just in case, look at how your site works for different platforms / countries / browsers - malware can only work for a specific category of users, this is done in order to go unnoticed.

so

It is necessary to check the above signs and places at least once a week, in order not to let the situation down if the site has been hacked. Tracking new software updates and important security updates along with scheduling backups and periodically changing passwords should generally be part of standard periodic procedures.

About what to do when you have already been hacked, we will talk in one of the following articles.