After purchasing a virtual and / or dedicated server, it is important to receive timely information about the unavailability of the service, that is, to monitor the main subsystems of the website.

A reliable website should be easily accessible to users 7 days a week, so it should be constantly checked for availability and performance:
')

• regular self-testing of the website using free tools ;
• constant monitoring of the availability of the site, which is optimal for the time interval: the majority of users will try to return to the site within 1-2 hours, more frequent checks do not guarantee a more prompt correction of problems faster than within an hour;
• monitoring of project components and analysis of application metrics : response speed, component errors, services, database response speed, non-indexed or slow query monitoring;
• website performance monitoring : slow loading of pages can cost customers losing, while monitoring results in timely notification of problems, allowing them to be eliminated promptly and minimized consequences;
• Monitoring of problems consists in tracking several parameters of a site with a frequency of at least once a minute from several geographic points in order to maximally cover the minute interval with checks and track possible problems related to the geography of users.

Among the possible verification criteria are the following problems:

• with a DNS server (when the site address cannot be determined at certain intervals, although the site itself is physically accessible);
• with a long response time (when updating the cache, for example, or when performing "heavy" tasks on the server side);
• with the planned execution of tasks (as a result of which the site will not be available only at certain points in time);
• with long latency times for static files (for example, due to network infrastructure or problems with physical media);
• with a database connection .

Many external services now provide detailed information about problems, up to error logs on the client side (with appropriate configuration and maintenance of error logs from the server). Such methods are especially good when you need to catch some kind of "floating" error - when you turn on the detailed logs of the error that occurs on the server side, you can effectively track and fix it.

Site work at the weekend / holiday season

There is a task: the website / server / service should work continuously for several days without human intervention. What can go wrong?

The usual malfunctions from time to time happen by themselves. Only here the night crash from Tuesday to Wednesday is decided by reloading the backup on Wednesday morning. And on weekends there are frequent failures "from Friday to Monday." How much a site can be in such a case during holidays depends on the length of holidays of responsible employees.

In general, the site is not good, but on weekdays the problem is solved quickly. How much time would the decision take in May if it were not for monitoring? Instead of a couple of hours it could be a couple of days, and this is not uncommon.

Do not make major code changes before long weekends. It is necessary to thoroughly test the system, as amended, in order for the changes to work properly. It is recommended to postpone any major changes until the website has less traffic load.

In addition to the usual problems, sites during the long absence of vigilant guards also like to catch other illnesses. For example, a domain or certificate may expire. Or decide to grow fat database. Or he can please the DNSBL or Roskomnadzor lists.

DNSBL blacklists

An important function is to check domains in DNSBL blacklists (DNS blacklist or DNS blocklist) - lists of hosts stored using the DNS architecture system. Usually used to combat spam.

These lists are independent and are formed each according to their own algorithm, because of which, as a result of a random error, even a harmless site may turn out to be there. An IP address from your subnet can be used for malicious purposes, for example, by spammers or other intruders, as a result of which the entire subnet may be blocked in the black list of the corresponding structure.

How does this threaten you? Letters from you will cease to come to clients, the site will become worse to appear in search engines, and so on in increasing terms. Therefore, the function of control and notification of hit in the most popular black lists is very popular.

Each administrator can configure their web server so that, for example, it does not receive emails from the servers listed in a specific list. It helps to fight spam, the spread of malware, DDoS attacks and other problems.

Online DNSBL blacklists, for example, antispamsniper.com or syslab.ru , allow you to filter spam using DNS to access spam IP addresses databases.

To check the presence of the specified IP address in the blacklists, enter the IP address (your current IP address is specified by default) and click the Check button.

DDoS protection

If a profit depends on the availability of a website, then you should prepare it for increasing loads (for example, during seasonal sales or Black Friday) and possible attacks by competitors and / or attackers who are counting on an increase in the response time of the website to incoming requests or partial / complete inaccessibility.

Technical planning

The server software on which the website or other resource is built must be periodically updated.

Planning technical work allows you to achieve two goals: not to send error notifications and not to record errors during a certain time interval in the statistics. At the same time, checks during technical work still go on and are regularly written to the log, and therefore can be useful for administrators: the log allows you to determine how long the update or reboot lasted, what errors were issued, what problems were observed and so on .

It is recommended to carry out planned work during a noticeable reduction (reduction) of client traffic, as well as during the absence of peak bandwidth.

Monitoring domain expiration and SSL certificate

Problems with the renewal of domains and certificates occur even in large companies. Therefore, the notification (via SMS or e-mail) that this domain needs to be renewed is extremely useful. For example, ping-admin.ru provides paid services for monitoring the results of monitoring.

Check domain expiration

You can check the domain for free using the nic.ru service.

You can check the domain validity time for free using the Whois Service .

Validate SSL certificate validity

Run the following command from a Linux command line to find out the validity of an SSL certificate using openssl:

$ echo | openssl s_client -servername  -connect : 2>/dev/null | openssl x509 -noout -dates 

In addition to the validity period, the SSL certificate contains a lot of interesting information. Each SSL certificate contains information about who issued it, to whom it was issued, its validity and so on.

All this data can be extracted from an SSL site certificate using the openssl program from the command line on Linux.

Check who issued the SSL certificate:

 $ echo | openssl s_client -servername site.com -connect site.com:443 2>/dev/null | openssl x509 -noout -issuer issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 

Check who issued the SSL certificate:

 $ echo | openssl s_client -servername site.com -connect site.com:443 2>/dev/null | openssl x509 -noout -subject subject= /CN=www.site.com 

Show all the above SSL certificate information with one command:

 $ echo | openssl s_client -servername site.com -connect site.com:443 2>/dev/null | openssl x509 -noout -issuer -subject -dates issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 subject= /CN=www.site.com notBefore=Mar 18 10:55:00 2017 GMT notAfter=Jun 16 10:55:00 2017 GMT 

Monitoring site availability

For effective work of any visited website, the constant availability of its materials for visitors is necessary, as well as the opportunity for the project administrator to have access to the server part to make changes or any other actions.

You can easily check the availability of the site from the command line in Linux and get the code from the server with the HTTP status using commands such as TELNET or CURL.

Website availability with CURL

Run the following command to check the availability of the site and receive a status message from the server:

 $ curl -Is http://www.site.com | head -1 HTTP/1.1 200 OK 

The status code '200 OK' means that the request was successfully completed and the site is available.

Here is another example that shows how curl displays different server responses:

 $ curl -Is http://site.com | head -n 1 HTTP/1.1 301 Moved Permanently 

Also, using curl, you can check the availability of a separate page on the site, for example:

 $ curl -Is http://www.site.com/en/Bash-Colors | head -n 1 HTTP/1.1 200 OK 

Website availability with TELNET

You can also check the availability of the site and receive status messages from the server using the telnet command:

 $ telnet www.site.com 80 Trying 91.206.200.119... Connected to www.site.com. Escape character is '^]'. HEAD / HTTP/1.0 HOST: www.site.com < ENTER> < ENTER> 

The conclusion that means that the site is available will look like this:

 HTTP/1.1 200 OK Server: nginx/1.1.10 Date: Sun, 26 May 2017 19:29:46 GMT *** 

In conclusion, I would like to note that there is always an option to write your own script to check uptime in PHP or Perl, or you can create a telegram bot to send notifications, but by calculating the daily income from websites and correlating it with the cost of monitoring, it is often cheaper to use paid services type PagerDuty .

Useful articles with reviews of monitoring services:

• How to find out the site uptime?
• 7 services for monitoring sites
• 7 more services for monitoring sites
• Introduction to Riemann
• Service Monitoring with Prometheus
• Monitoring: shellhacks.com blog
• Netdata: real-time monitoring
• Security Monitoring with Sysdig Falco