What is the monitoring of site availability? Service Overview HostTracker
HostTracker Company is one of the market leaders in the provision of site monitoring services and various web services. It has a variety of different tools that are constantly being improved, taking into account the trends of the time. How did the company come about? Why is there a need for monitoring? How can he help in the development and testing of sites and servers? This review will reveal the essence of the questions, as well as show how a simple desire to optimize your work can be transformed into a useful product for others, as well as not losing the spirit of startup and always keep pace with your customers.
Everyone involved in IT is familiar with the feeling when a seemingly perfectly configured server (site, base, program, network) suddenly stops working as it should. The search for a problem may take a long time, during which, of course, this disgrace will continue. Or occur regularly. Or it will not happen - but this only makes it worse: for when will this incomprehensible something come out another time? And if at the time of the transfer of the project to the client? Somewhere from such reflections and HostTracker appeared. His founding father sought good work from his main activities, and decided to automate the monitoring process. If at that time (the stern beginning of zero) there were reliable monitoring services, it is quite possible, HostTracker would never have appeared. And so I had to write a simple script to check my zoo sites. But the trouble is - a simple script had the same problems that were mentioned at the beginning of this paragraph. He did not always work, and not always - correctly. I had to optimize the script a little and make it more reliable, including by duplicating and distributing on several machines. After a series of optimizations, the idea came that this could be useful to someone else, and HostTracker became public. And after the emergence of many clients - the idea to monetize the effort. That is, our service appeared to solve purely personal tasks, but in the process of development it became available to others to help them solve the same problems that they once faced us. Not everyone has the time to write a simple script. And then add a little more. And then a little optimize.
This part of the review will tell you about the most popular feature - a regular check to immediately identify any problems with a site or other entity accessible from the network. These checks can work using different protocols: http (s), icmp (aka ping), port (checking any port using TCP), and some others. For a start, we note that there are different approaches to monitoring. There is internal monitoring - when the state of the site is assessed by software hosted on the same server. Conventionally, you can also include such tools as Yandex.Metrica - the script embedded in the page code sends information about customers who have visited the site, and can draw indirect conclusions about the performance of the site. The other type of monitoring is external, or, as the lawyers put it, “monitoring by a third party”. It consists in simulating a visit to the site by real users from real addresses, and draws conclusions from the actual responses of the server to the sent requests. It is about this method that will be discussed further.
')
Most simply, it can be divided into two components: "internal" and "external." The first is located in the cloud and consists of computing servers, databases, file storage and other components. All of this is owned and operated directly by HostTracker employees. The second component is nodes, or agents, which are located throughout the world and act as independent monitoring servers, like a “secret customer” in stores. They perform checks on sites and other scanned objects by creating standard queries for the protocol being checked. If this is a website check, then regular http requests are created that are no different from requests from real visitors to the site. These servers are completely independent from each other and from the internal component of the system. All they do is get a list of sites from the central server for testing, and give him the result of the checks. Some of these servers are rented by HostTreker, but most are owned by partners who provide them free of charge or with a significant discount in exchange for back links or discounts on our services. Or just a friendship.
This distribution of infrastructure is due to the operation of the monitoring service.
Regular checks occur at a predetermined customer interval, starting at 1 minute. When the verification time comes, the server sends the task to one, randomly determined agent - part of the HostTrecker external network. It performs a check on the specified protocol and sends back the response of the server being checked. These results are already processed on the inside. If the answer is positive - nothing happens, just an entry appears in the database (it is available to the client as a check log). If an error is suddenly noticed, then tasks for checking the same site are immediately created by several more, again, randomly selected servers from the common network. Then the answer of all these servers is already analyzed. If the error is not confirmed by most servers, then it remains in the log, but peace of mind remains. Anything can happen: network lag, query overload, habraeffect, after all. If most or all of the agents confirm the error, then an assumption is made that the trouble has come and the problem is more serious - most likely, at this moment many people cannot get to the site. And for almost all commercial sites this is a direct loss of money.
Depending on the settings, after identifying the problem, many different things can occur. Immediately, a primary diagnosis of the problem occurs: an error code is recorded, if any, or a protocol response. From these data, it is often possible to speculate about the causes of the error: network failure, server error, or anything else. But first of all, as a rule, the client is interested in finding out about the problem. To do this, it is possible to register your phone, email or other communication channel to receive alerts from HostTreker. They can be sent to different people, to different contacts and at different times. And there are many interesting features. Of course, the message can be sent immediately, which is most often done. But it is difficult to satisfy a demanding customer. First, it turns out that there are “unimportant” websites, because of the short-term downtime that it is ugly to wake a respected admin. Secondly, there are “important” people who are ugly disturbed because of every sneeze. Therefore, HostTracker offers an alert escalation feature - send an alert to a specific address only some time after the site crashes, if it has not recovered by that time. Also, for each contact, it is possible to create a “work schedule” - to set a time interval during which alerts can be sent to it. At another time, HostTracker will not disturb this person. This also turned out to be convenient for some clients. For example, this configuration will provide an alert to this person from 8 to 19 hours on workdays in cases where the site is “lying” an hour or more:
In further publications, we describe the many functions of the service that have accumulated to date. In parallel, the stories of their appearance and development, examples of use and many other things from our humble experience will be mentioned. Now let me say that one of the distinguishing features of the service is customer orientation. Many service functions were developed at the request of customers, friends and partners, which allowed them to solve specific technical problems, the solution of which on their own would require much more resources. Therefore, we always welcome feedback and your suggestions!
Introduction, or startup history
Everyone involved in IT is familiar with the feeling when a seemingly perfectly configured server (site, base, program, network) suddenly stops working as it should. The search for a problem may take a long time, during which, of course, this disgrace will continue. Or occur regularly. Or it will not happen - but this only makes it worse: for when will this incomprehensible something come out another time? And if at the time of the transfer of the project to the client? Somewhere from such reflections and HostTracker appeared. His founding father sought good work from his main activities, and decided to automate the monitoring process. If at that time (the stern beginning of zero) there were reliable monitoring services, it is quite possible, HostTracker would never have appeared. And so I had to write a simple script to check my zoo sites. But the trouble is - a simple script had the same problems that were mentioned at the beginning of this paragraph. He did not always work, and not always - correctly. I had to optimize the script a little and make it more reliable, including by duplicating and distributing on several machines. After a series of optimizations, the idea came that this could be useful to someone else, and HostTracker became public. And after the emergence of many clients - the idea to monetize the effort. That is, our service appeared to solve purely personal tasks, but in the process of development it became available to others to help them solve the same problems that they once faced us. Not everyone has the time to write a simple script. And then add a little more. And then a little optimize.
Monitoring: the essence
This part of the review will tell you about the most popular feature - a regular check to immediately identify any problems with a site or other entity accessible from the network. These checks can work using different protocols: http (s), icmp (aka ping), port (checking any port using TCP), and some others. For a start, we note that there are different approaches to monitoring. There is internal monitoring - when the state of the site is assessed by software hosted on the same server. Conventionally, you can also include such tools as Yandex.Metrica - the script embedded in the page code sends information about customers who have visited the site, and can draw indirect conclusions about the performance of the site. The other type of monitoring is external, or, as the lawyers put it, “monitoring by a third party”. It consists in simulating a visit to the site by real users from real addresses, and draws conclusions from the actual responses of the server to the sent requests. It is about this method that will be discussed further.
')
Infrastructure, or how it works
Most simply, it can be divided into two components: "internal" and "external." The first is located in the cloud and consists of computing servers, databases, file storage and other components. All of this is owned and operated directly by HostTracker employees. The second component is nodes, or agents, which are located throughout the world and act as independent monitoring servers, like a “secret customer” in stores. They perform checks on sites and other scanned objects by creating standard queries for the protocol being checked. If this is a website check, then regular http requests are created that are no different from requests from real visitors to the site. These servers are completely independent from each other and from the internal component of the system. All they do is get a list of sites from the central server for testing, and give him the result of the checks. Some of these servers are rented by HostTreker, but most are owned by partners who provide them free of charge or with a significant discount in exchange for back links or discounts on our services. Or just a friendship.
This distribution of infrastructure is due to the operation of the monitoring service.
Algorithm of the service
Regular checks occur at a predetermined customer interval, starting at 1 minute. When the verification time comes, the server sends the task to one, randomly determined agent - part of the HostTrecker external network. It performs a check on the specified protocol and sends back the response of the server being checked. These results are already processed on the inside. If the answer is positive - nothing happens, just an entry appears in the database (it is available to the client as a check log). If an error is suddenly noticed, then tasks for checking the same site are immediately created by several more, again, randomly selected servers from the common network. Then the answer of all these servers is already analyzed. If the error is not confirmed by most servers, then it remains in the log, but peace of mind remains. Anything can happen: network lag, query overload, habraeffect, after all. If most or all of the agents confirm the error, then an assumption is made that the trouble has come and the problem is more serious - most likely, at this moment many people cannot get to the site. And for almost all commercial sites this is a direct loss of money.
Error handling and customer notification
Depending on the settings, after identifying the problem, many different things can occur. Immediately, a primary diagnosis of the problem occurs: an error code is recorded, if any, or a protocol response. From these data, it is often possible to speculate about the causes of the error: network failure, server error, or anything else. But first of all, as a rule, the client is interested in finding out about the problem. To do this, it is possible to register your phone, email or other communication channel to receive alerts from HostTreker. They can be sent to different people, to different contacts and at different times. And there are many interesting features. Of course, the message can be sent immediately, which is most often done. But it is difficult to satisfy a demanding customer. First, it turns out that there are “unimportant” websites, because of the short-term downtime that it is ugly to wake a respected admin. Secondly, there are “important” people who are ugly disturbed because of every sneeze. Therefore, HostTracker offers an alert escalation feature - send an alert to a specific address only some time after the site crashes, if it has not recovered by that time. Also, for each contact, it is possible to create a “work schedule” - to set a time interval during which alerts can be sent to it. At another time, HostTracker will not disturb this person. This also turned out to be convenient for some clients. For example, this configuration will provide an alert to this person from 8 to 19 hours on workdays in cases where the site is “lying” an hour or more:
In further publications, we describe the many functions of the service that have accumulated to date. In parallel, the stories of their appearance and development, examples of use and many other things from our humble experience will be mentioned. Now let me say that one of the distinguishing features of the service is customer orientation. Many service functions were developed at the request of customers, friends and partners, which allowed them to solve specific technical problems, the solution of which on their own would require much more resources. Therefore, we always welcome feedback and your suggestions!
Source: https://habr.com/ru/post/307462/
All Articles