Search by torrents in the DHT network
There are few people among users and readers of Habr who would not use torrents. And surely, many people understand the vulnerability of the central links of this system - torrent trackers (stories from Pirate Bay and the former torrents.ru, now a root tracker, is a confirmation of this).
However, not everyone knows that there has long been an alternative to traditional trackers - the DHT network, which allows you to organize distributions without a tracker. Until now, it has not received much distribution due to the impossibility of searching within the network - the required file could be downloaded only if there is a distribution magnet link. However, the first and unique search service on the DHD network BTDigg.org has recently appeared.
Service with a very concise design sharpened for one function - search:
')
To reduce the dependence on trackers, in 2007, the developers of the bittorrent protocol adopted its modification on the basis of a DHT structure , which made it possible to do distributions without a tracker.
What is a DHT network? In fact, this is a distributed network of torrent clients that stores identifiers (hashes) of all public hands - the so-called. magnet-link, which is a 160-bit random number. Client DHT nodes are responsible for announcing their torrents and searching for the requested ones. Due to the fact that all modern clients support DHT, one can say that there are practically all public torrents on this network (the network "does not see private torrents and does not allow downloading). Practically - because DHT in the client can be disabled. The search is based on the meta-information that the torrent file contains - that is, simply speaking, it searches by the name of the torrent and by the names of the files included in it. The user opens the magnet link in his torrent client, he finds all the distributors of the file and then downloads as usual - the tracker is not involved in this thread.
The disadvantage of this solution is that DHT responds to only one request - which IP addresses have peers that distribute a torrent with this infohash. That is, you can start downloading via DHT only if you have the desired magnet link. In this sense, a DHT network resembles the Internet without Google: you can get to the desired page only if you know its exact address. Yes, the DHT network can be created to work without torrent trackers, but if they are closed, the file sharing process would be very complicated.
Therefore, it is obvious that this network has needed its own search service since its inception. However, despite the simplicity of the idea, in implementation it turned out to be a very difficult technical task. The creators of the service took about six months to realize that it was possible to do this at all.
This is NOT a search service for torrent trackers. The functionality of BTDigg.org is much more powerful: this is the first and so far the only search engine on the DHT network - that is, in fact, for all public torrents on the Internet. As the source data, requests of info-hashes from other nodes are used, which are received from their own DHT cluster. Further, after clearing from the markups, these torches get torrents from peers participating in file sharing. The contents of the torrents themselves are not downloaded at any stage.
The creators of the resource declare respect for the community, so they do not use any prohibited methods of data collection (which allow increasing the speed of collecting information, but at the same time lead to an additional load on other members of the network). Their 24x7 cluster supports the DHT network. BTDigg analyzes the activity of distributions, indexing it once a day (the most popular distributions from the top 100 are in real time, every 10 seconds). The search supports the search queries language Sphinx (unfortunately, I did not find any language descriptions in Russian).
In the search results are information that contains a torrent file - that is, its name and size. The output can be sorted by relevance and number of peers (that is, customers who put this file to download).
Of the additional features so far only - the top 100 torrents for 2 weeks and real-time top 100 for the last 10 minutes, which is updated every 10 seconds without reloading the page.
In addition, the service also provides an API of its functionality - both to calculate the popularity of torrents in the network, and search (currently used in the qBittorrent client). The API is open, but has limitations on the number of requests per unit of time.
Due to the technical complexity of the task, the project started quite recently - only in January 2011. Now its daily audience is 10,000 people, but the potential, of course, is much higher. The project is not very well known in runet (on Habré there was only a back post in the sandbox some time ago), but in its international team Russian participation was clearly expressed - at least I spoke with its Russian-speaking representative with the rare name John Smith. More details about themselves, for obvious reasons, they do not disclose.
Speaking of copyright holders (who did not understand, they are the “understandable reason”). I give the word to John:
Therefore, so far, in order not to give the right holders the opportunity to negotiate in a vein that is different from the correspondence intellectual discussion - well, there, complaints about the hoster abuse, seizure of servers, abuse in the media, etc. - they are forced to remain anonymous, although this complicates the development of the project.
Many new features are being tested and soon on the site, including setting up and filtering out the issue, as well as improving network coverage - BTDigg.org will be larger (even very rare torrents) and will be faster to index new distributions in the DHT network.
Separate greetings to the Russian audience - the creators of the service are planning to make Russian localization, and the support now understands Russian. There are no plans to monetize the service yet - perhaps an advertisement will appear if the audience is not against it.
However, not everyone knows that there has long been an alternative to traditional trackers - the DHT network, which allows you to organize distributions without a tracker. Until now, it has not received much distribution due to the impossibility of searching within the network - the required file could be downloaded only if there is a distribution magnet link. However, the first and unique search service on the DHD network BTDigg.org has recently appeared.
Service with a very concise design sharpened for one function - search:
')
Torrent hash search engine
To reduce the dependence on trackers, in 2007, the developers of the bittorrent protocol adopted its modification on the basis of a DHT structure , which made it possible to do distributions without a tracker.
What is a DHT network? In fact, this is a distributed network of torrent clients that stores identifiers (hashes) of all public hands - the so-called. magnet-link, which is a 160-bit random number. Client DHT nodes are responsible for announcing their torrents and searching for the requested ones. Due to the fact that all modern clients support DHT, one can say that there are practically all public torrents on this network (the network "does not see private torrents and does not allow downloading). Practically - because DHT in the client can be disabled. The search is based on the meta-information that the torrent file contains - that is, simply speaking, it searches by the name of the torrent and by the names of the files included in it. The user opens the magnet link in his torrent client, he finds all the distributors of the file and then downloads as usual - the tracker is not involved in this thread.
The disadvantage of this solution is that DHT responds to only one request - which IP addresses have peers that distribute a torrent with this infohash. That is, you can start downloading via DHT only if you have the desired magnet link. In this sense, a DHT network resembles the Internet without Google: you can get to the desired page only if you know its exact address. Yes, the DHT network can be created to work without torrent trackers, but if they are closed, the file sharing process would be very complicated.
Therefore, it is obvious that this network has needed its own search service since its inception. However, despite the simplicity of the idea, in implementation it turned out to be a very difficult technical task. The creators of the service took about six months to realize that it was possible to do this at all.
What can BTDigg
This is NOT a search service for torrent trackers. The functionality of BTDigg.org is much more powerful: this is the first and so far the only search engine on the DHT network - that is, in fact, for all public torrents on the Internet. As the source data, requests of info-hashes from other nodes are used, which are received from their own DHT cluster. Further, after clearing from the markups, these torches get torrents from peers participating in file sharing. The contents of the torrents themselves are not downloaded at any stage.
The creators of the resource declare respect for the community, so they do not use any prohibited methods of data collection (which allow increasing the speed of collecting information, but at the same time lead to an additional load on other members of the network). Their 24x7 cluster supports the DHT network. BTDigg analyzes the activity of distributions, indexing it once a day (the most popular distributions from the top 100 are in real time, every 10 seconds). The search supports the search queries language Sphinx (unfortunately, I did not find any language descriptions in Russian).
In the search results are information that contains a torrent file - that is, its name and size. The output can be sorted by relevance and number of peers (that is, customers who put this file to download).
Of the additional features so far only - the top 100 torrents for 2 weeks and real-time top 100 for the last 10 minutes, which is updated every 10 seconds without reloading the page.
In addition, the service also provides an API of its functionality - both to calculate the popularity of torrents in the network, and search (currently used in the qBittorrent client). The API is open, but has limitations on the number of requests per unit of time.
About
Due to the technical complexity of the task, the project started quite recently - only in January 2011. Now its daily audience is 10,000 people, but the potential, of course, is much higher. The project is not very well known in runet (on Habré there was only a back post in the sandbox some time ago), but in its international team Russian participation was clearly expressed - at least I spoke with its Russian-speaking representative with the rare name John Smith. More details about themselves, for obvious reasons, they do not disclose.
Speaking of copyright holders (who did not understand, they are the “understandable reason”). I give the word to John:
So far, there have been no serious raids on us (unless some porn studios require that we exclude links to their content from the database). From the point of view of the law (in our opinion) we behave like an ordinary search engine (google, yandex, etc.), even more so - we don’t give direct links (a magnet link is not a link in its direct sense). But in the world of lawyers and copyright holders, the law can be turned in any direction and the example of pirate bay and rutracker is proof of that.
Therefore, so far, in order not to give the right holders the opportunity to negotiate in a vein that is different from the correspondence intellectual discussion - well, there, complaints about the hoster abuse, seizure of servers, abuse in the media, etc. - they are forced to remain anonymous, although this complicates the development of the project.
Development plans
Many new features are being tested and soon on the site, including setting up and filtering out the issue, as well as improving network coverage - BTDigg.org will be larger (even very rare torrents) and will be faster to index new distributions in the DHT network.
Separate greetings to the Russian audience - the creators of the service are planning to make Russian localization, and the support now understands Russian. There are no plans to monetize the service yet - perhaps an advertisement will appear if the audience is not against it.
Source: https://habr.com/ru/post/124496/
All Articles