Creating a search engine, or Automation Yandex.Server
I am from Belarus, here the largest Internet provider is byfly. This provider provides its users with free access to all sites that are hosted domestically (guest resources).
Each byfly user in the arsenal had a set of files with links to free resources available. So the idea of creating a search engine for these resources came, and already in August 2009, the first users saw it. Attendance grew rapidly and, at the peak of popularity, the resource was visited by about 34,000 unique users per day.
In the heart of the search engine running Yandeks.Server. This is how its control panel looks like:
')
It immediately became clear that having only this set of functionality (3 buttons is on / off search, on / off indexing and off Ya.Server), there is no sane search engine to do.
So I had to modify it a bit and it turned out like this:
Many additional features have been implemented, such as:
When an administrator in a new control panel performs an action (for example, wants to remove a site from the index), the system will generate a configuration file with the necessary parameters:
After that, it will send a request to update the index to the standard Yandex.Server control panel. Reindexing occurs according to the rules specified in the configuration file. The administrator will see the result of the operation in the modified control panel. All manipulations are performed in the background via ajax requests.
The peak of popularity per day was: more than 300,000 search queries, ~ 34,000 unique users, in the search index more than 2,000 sites. For all needs there is always more than enough hardware VPS Intel 2.33 (1 core), 1 GB of RAM.
Users themselves on the enthusiasm began to develop various add-ons for browsers with search and even the desktop version. Others, so much wanted to get into the search index, that even threatened :-)
Later, byfly abolished free access and popularity gradually began to decline. Now it is kept at the level of 3,000 unique users per day.
search.pusk.by was originally calculated as a short-term project. I estimated the time of his life at 1 year, but he lasted longer :-)
Each byfly user in the arsenal had a set of files with links to free resources available. So the idea of creating a search engine for these resources came, and already in August 2009, the first users saw it. Attendance grew rapidly and, at the peak of popularity, the resource was visited by about 34,000 unique users per day.
In the heart of the search engine running Yandeks.Server. This is how its control panel looks like:
')
It immediately became clear that having only this set of functionality (3 buttons is on / off search, on / off indexing and off Ya.Server), there is no sane search engine to do.
So I had to modify it a bit and it turned out like this:
Many additional features have been implemented, such as:
- splitting the index into resources and creating for each of them their own mask and indexing rules;
- automatic index update system;
- a system for tracking abuse by webmasters;
- automatic index cleaning system of garbage documents;
- separate searches for videos and abstracts;
- health monitoring system ...
When an administrator in a new control panel performs an action (for example, wants to remove a site from the index), the system will generate a configuration file with the necessary parameters:
function cfg_file_generate($adr, $option, $timeout, $delay, $rules) { global $cfg_useragent; $cfg = "<Webds> $rules <IndexedArea> HttpPrefix http://$adr Options $option <HttpOptions> Timeout $timeout Delay $delay <Headers> User-Agent: $cfg_useragent </Headers> </HttpOptions> </IndexedArea> </Webds>"; return $cfg; } After that, it will send a request to update the index to the standard Yandex.Server control panel. Reindexing occurs according to the rules specified in the configuration file. The administrator will see the result of the operation in the modified control panel. All manipulations are performed in the background via ajax requests.
The peak of popularity per day was: more than 300,000 search queries, ~ 34,000 unique users, in the search index more than 2,000 sites. For all needs there is always more than enough hardware VPS Intel 2.33 (1 core), 1 GB of RAM.
Users themselves on the enthusiasm began to develop various add-ons for browsers with search and even the desktop version. Others, so much wanted to get into the search index, that even threatened :-)
Later, byfly abolished free access and popularity gradually began to decline. Now it is kept at the level of 3,000 unique users per day.
search.pusk.by was originally calculated as a short-term project. I estimated the time of his life at 1 year, but he lasted longer :-)
Source: https://habr.com/ru/post/146621/
All Articles