Who indexed my site?

Creating a website on Biggo , it is possible to take advantage of another SaaS advantage - accounting for site indexing by search engines. Ordinary JS-counters or image-based counters, of course, do not provide such an opportunity. In liveinternet there is an opportunity to take into account the requests of search bots, but it is given there to "catch up" in the express audit section. In addition, in order for this report to work, you need to install the proposed PHP code.

Our system independently collects data to create this necessary for webmasters and optimizers report and displays the results in the form of beautiful interactive graphs

Pie chart showing the shares of different search robots


Robot indexing chart for the last month


The index chart of the selected robot for the last month and the pages indexed today. You can get into this report by clicking on the desired part of the pie chart.
')
We have been collecting statistics for a long time, but we started to make reports recently, we decided to do this first, because it will perfectly complement the information that can be obtained using public meters.

Technical features

The data is collected in the MongoDB collection, then they are processed once an hour and converted into a convenient form of graphing:
{ "_id" : ObjectId("4c4ec447e555c04d1e002fbf"), "bots" : [
{
"num" : 2,
"bot_name" : "Google"
},
{
"num" : 1,
"bot_name" : "Nigma"
}
], "bdate" : "2010-07-15", "site_id" : 178 }
{ "_id" : ObjectId("4c4ec448e555c04d1e002fc2"), "bots" : [ { "num" : 10, "bot_name" : "Yandex" } ], "bdate" : "2010-07-15", "site_id" : 250 }
{ "_id" : ObjectId("4c4ec448e555c04d1e002fc4"), "bots" : [ { "num" : 3, "bot_name" : "Twiceler" } ], "bdate" : "2010-07-15", "site_id" : 107 }
{ "_id" : ObjectId("4c4ec446e555c04d1e002fa4"), "bots" : [
{
"num" : 6,
"bot_name" : "Google"
},
{
"num" : 10,
"bot_name" : "Yahoo!"
},
{
"num" : 9,
"bot_name" : "Yandex"
},
{
"num" : 5,
"bot_name" : "MSN"
},
{
"num" : 2,
"bot_name" : "Twiceler"
},
{
"num" : 1,
"bot_name" : "Rambler"
}
], "bdate" : "2010-07-15", "site_id" : 133 }

In this format, it is already a matter of engineering to build a schedule by day - this is efficient and convenient.

MongoDB is ideally suited for this task for at least 3 reasons:

• Allows you to store attached objects
• Allows you to set the capped parameter for the collection, so old data does not need to be deleted, it will be overwritten by new
• Does not load the main postgres base, which has a lot of cases

Charting Client Library - Open Flash Chart 2