Sensors for monitoring content on a web server
Reflecting on the “cheaper” monitoring of updates to the necessary web pages, he formulated the concept of “sensors”. I did not see any references earlier, so I publish here and now.
The essence of the concept lies in a special revision of the Web server software, i.e. in the creation and connection of a software module that will “monitor” for incoming HTTP requests and “listen” to the server's responses. Each request (on the basis of http-headers information, a unique identifier of the requested resource can be compiled) must be matched with a response (can be processed using the md5 algorithm or similar) of the server. In the next step, this pair is searched for a database of meta-descriptions of resources already accumulated by the Web server. When a change is detected, it is suggested immediately (or as the Web server is unloaded) to send a certain meta-description of the found changes to the main public server (for example, using a GET request, in which meta-information will be encoded as parameters). Actually, therefore (the analogy with various sensors used in technical practice is obvious) I decided to call the module “sensor”. And then, already from the public server, information about updating the page or any other content resource can be obtained by any “consumer” on request or by other means. I think the idea is clear.
This concept was already relatively long ago (in 2005) presented in his dissertation . True, the dissertation accent was on creating a system for modeling the process of monitoring websites and therefore the concept of sensors was not really discussed. Now, while I am sitting without work, I decided to include the habrosoal community in my research and I hope to get an interesting response (I could not reach the interested people in due time). Thanks in advance for participating in the discussion!
')
PS: My simulation experiment showed greater efficiency (read optimality in terms of “freshness” and “traffic saving” parameters) of a monitoring system based on sensors rather than a classical monitoring system (for example, search engine robots).
The essence of the concept lies in a special revision of the Web server software, i.e. in the creation and connection of a software module that will “monitor” for incoming HTTP requests and “listen” to the server's responses. Each request (on the basis of http-headers information, a unique identifier of the requested resource can be compiled) must be matched with a response (can be processed using the md5 algorithm or similar) of the server. In the next step, this pair is searched for a database of meta-descriptions of resources already accumulated by the Web server. When a change is detected, it is suggested immediately (or as the Web server is unloaded) to send a certain meta-description of the found changes to the main public server (for example, using a GET request, in which meta-information will be encoded as parameters). Actually, therefore (the analogy with various sensors used in technical practice is obvious) I decided to call the module “sensor”. And then, already from the public server, information about updating the page or any other content resource can be obtained by any “consumer” on request or by other means. I think the idea is clear.
This concept was already relatively long ago (in 2005) presented in his dissertation . True, the dissertation accent was on creating a system for modeling the process of monitoring websites and therefore the concept of sensors was not really discussed. Now, while I am sitting without work, I decided to include the habrosoal community in my research and I hope to get an interesting response (I could not reach the interested people in due time). Thanks in advance for participating in the discussion!
')
PS: My simulation experiment showed greater efficiency (read optimality in terms of “freshness” and “traffic saving” parameters) of a monitoring system based on sensors rather than a classical monitoring system (for example, search engine robots).
Source: https://habr.com/ru/post/47655/
All Articles