Upgrade
Hi oll!
Surely, many of you have noticed that recently attempts to control the Internet are getting stronger and stronger, there are many examples - from closing torrents.ru without trial and investigation (and they cooperate with copyright owners !!) to demonstration trapping of various hackers, what would others have been quieter. Yeah. Fuck you.
I propose to organize and start developing our new (national, hehe) data exchange system, which will be the development of existing torrents, him and others.
')
Immediately I say, probably, all this is somehow implemented in various information exchange tools - the goal is to make it more accessible, more convenient and more alive.
Caution - a lot of text without pictures.
I will simply try to offer something of my own, but I will be happy with any additions to my thoughts. After all, I can not be right in the root, right?)
At the moment (this is only an idea and everything can change abruptly), it is believed that XMPP is very good and use it for client and server interaction to use some (well, probably BitTorrent) technology to exchange data directly.
Accordingly, what we have at the base:
For the basics, this is enough for the eyes and the ears — you can implement almost anything your heart desires, and therefore let's try to solve problems with the help of this toolkit.
Here I propose to just sit and think about the structuring of distributions and tph. At the moment, everything is on a rutreker and many places are still stored, for example, in BB codes and tph, and this is often not very convenient for machines) However, they still need to fill out a form when creating a distribution on a rutreker. So why not make it all more perfect and not store the structure instead of plain text?
For this, I see the sufficiency of introducing digital signatures under various content (mostly - .torrent files, etc.). By this signature you can understand what kind of person in the network checked it and, in fact, if anything - questions to him. Perhaps there is an option to organize a multi-level signature - say, first the small moderator signs, and then it must be signed by someone more responsible, etc. or you can organize in such a way the levels of confidence in the content.
To protect against viruses, there is the next idea - to make just bots with which you will send content for a virus scan and they will sign this content if everything has passed the test.
When you enter “House MD” in the search bar on the rutracker, you just get a hundred different hands ... it is sad and you have to sit and choose what you would be comfortable and still look at the heading, like quality, but then it turns out that it’s not season, but not always a good idea to enter in the search bar also "Season 3" - you can again miss something.
You just need to create one distribution for everything and be able to select the .torrent file you need in one place and do not rummage through it all. You can put all sorts of tools nearby - what kind of subtitles are needed, what translation, what season, what quality of the picture, etc.
In order to be able to merge distributions you just need to separately sign them all - when there is some kind of distribution, and then more content appears, then it adds its file to the already existing one and it queues for moderation and this new content ( not the entire distribution) will be marked “not checked” (well, or as it will be better) until it is signed by the moderator and also everything as in paragraph 2. you just need to sign all of this separately and separately implement the functionality of clause 2. Looks nice.
It's more complicated. It is necessary to increase the viability of the network - for this you need to implement the following algorithm to search for a new tracker.
To do this, we need the following functionality:
The first mechanism is needed in order to understand what to connect when the tracker falls, and the second if the current tracker is working, but does not want to handle this client.
With the second, everything is simple and understandable - just at announce, to return something like “redirect to ****” to it and that's it.
But for the first you need a special server setting - you need to create some network of servers (let it be a nexus), it is desirable that they are geographically distant. Each server from this nexus will have a prioritized list of this entire small network, which it will send to all its clients. If the tracker suddenly stops responding, clients will go down the list of these servers and connect somewhere. At this time, data is being shared inside the network and determining who will actually distribute the content now (this will happen very quickly - the network is small and just “agree”, and for implementation you can estimate some heuristics and come up with some simple and unambiguous rules to determine the successor of the deceased tracker), and then after you decided, then redirect all of this new tracker.
This should not cause any configuration problems - people already set up backups of their torrents, and here you just need to talk to people and almost don’t need to configure the server - they will do everything themselves.
It is still important to remain anonymous and make this network more distributed. Here it is proposed to make it through the complication of the hierarchy, but the complication should be such that it only strengthens the network, rather than weakens it.
Our trackers are already merged into nexus and can form with each other some zone that can function as long as at least one member of the nexus is alive and all the content from the dead members of the nexus is stored accordingly. Yes, there is a problem with the load, but if there are 20 servers in a nexus, then there probably will never be such a problem.
If it turns out that the load is too large, then you can always raise the support server and redirect some of the clients, etc. to it.
In order to identify the server to participate in the nexus, digital signatures will be used, etc. This will protect the network from external influences. And the subsidiary subsidiary servers may have a different key and, accordingly, when creating a nexus, everyone exchanges a pair of keys - from the main server and from the subsidiary ones. Auxiliary can be as many as you want, but only one main one. This mechanism was created to simplify the connection and expansion of the network by one network participant - so that it would not be necessary for everyone to communicate and exchange keys. In fact, secondary servers are only caching proxies. But in general, their need is rather illusory.
This is very easy to implement through the answer to announce, so we will not be distracted by it.
To do this, you can simply implement for each user his own section, for distribution of which he can give links that can not be opened without, for example, a generated password. In principle, there is also nothing particularly difficult here - trackers are always there and will work. In principle, you can not even store everything on the servers, but you can make the tracker directly in the client - so that the user can make requests to this client via the XMPP server for these private distributions.
No, not all, they are simply mass and over and above the goal (or over desire - as anyone) - is to create the perfect tool for the exchange of information. In addition to the above, there are a lot of introductions to all of this. For example, to create an anonymous semi-automatic method of merging information into the network (for a la wikiLeaks), to create opportunities for closed trackers (so that only some members of this super-network could participate in file sharing).
As for the interface, I want to make a single basic interface and I don’t want it to be a browser. The browser has too few features for real file sharing.
And in the pursuit of this - to have a certain hierarchy of Nexus, which would be clearly seen at the palm of your hand “the whole Internet”. For example, splitting on a territorial basis, on a direction (IT, music and tph), etc. And in the nexus themselves, too, create their own hierarchy, etc. I even think that it would be interesting to start with this. And m. so that the Nexus would not participate in the hierarchy, but the trackers themselves. It is likely that you will have to do something like DNS with the issuance of names in this hierarchy. But in general, there is already a lot to think about how to organize it, that there would be no cybersquatting, etc. Create the ability to create closed communities without your own tracker. But there is already a security issue - the server has all the information and in principle it can send everything to other network participants ... here you can use the “tracker in the client” mechanism with synchronization of network participants via an encrypted channel. In general, many, many good things.
In general, I propose to put together a small team and start creating it all. If not us, then others. People are already fussing at the expense of distributed different things (for example, the diaspora). All will be useful.
Waiting for all in the comments.
Surely, many of you have noticed that recently attempts to control the Internet are getting stronger and stronger, there are many examples - from closing torrents.ru without trial and investigation (and they cooperate with copyright owners !!) to demonstration trapping of various hackers, what would others have been quieter. Yeah. Fuck you.
I propose to organize and start developing our new (national, hehe) data exchange system, which will be the development of existing torrents, him and others.
')
Immediately I say, probably, all this is somehow implemented in various information exchange tools - the goal is to make it more accessible, more convenient and more alive.
Caution - a lot of text without pictures.
Major current network issues
- Lots of duplicate content
- Content is not always proven and often zavusovannym
- Several different content formats are scattered across the network (we often find what we need, but not in the format we would like to).
- For more or less fast jumps, a tracker is still required and if it stops working, then the speed starts to fall. (Personally, my DHT generally puts the router pretty quickly and therefore it is disabled on all home machines)
- At the moment, it is not so easy to distribute left content on torrents. And often it is completely closed
- There is no possibility to automatically update torrents (for example, for TV shows this is very important)
- The difficulty of initiating the distribution of content without using a tracker (if I just want to give a friend a file in p2p, then this just won't work)
What can we do about it?
I will simply try to offer something of my own, but I will be happy with any additions to my thoughts. After all, I can not be right in the root, right?)
General ideology
At the moment (this is only an idea and everything can change abruptly), it is believed that XMPP is very good and use it for client and server interaction to use some (well, probably BitTorrent) technology to exchange data directly.
Accordingly, what we have at the base:
- a set of servers that can communicate with each other
- a set of clients that can exchange information with any of the servers and with any of the clients (but already through the server)
- the information itself can be absolutely any
For the basics, this is enough for the eyes and the ears — you can implement almost anything your heart desires, and therefore let's try to solve problems with the help of this toolkit.
Problem 1 - Content Duplication
Here I propose to just sit and think about the structuring of distributions and tph. At the moment, everything is on a rutreker and many places are still stored, for example, in BB codes and tph, and this is often not very convenient for machines) However, they still need to fill out a form when creating a distribution on a rutreker. So why not make it all more perfect and not store the structure instead of plain text?
Problem 2 - Content is not always proven and often zavirusovannym
For this, I see the sufficiency of introducing digital signatures under various content (mostly - .torrent files, etc.). By this signature you can understand what kind of person in the network checked it and, in fact, if anything - questions to him. Perhaps there is an option to organize a multi-level signature - say, first the small moderator signs, and then it must be signed by someone more responsible, etc. or you can organize in such a way the levels of confidence in the content.
To protect against viruses, there is the next idea - to make just bots with which you will send content for a virus scan and they will sign this content if everything has passed the test.
Problem 3 - Several different content formats scattered across the network.
When you enter “House MD” in the search bar on the rutracker, you just get a hundred different hands ... it is sad and you have to sit and choose what you would be comfortable and still look at the heading, like quality, but then it turns out that it’s not season, but not always a good idea to enter in the search bar also "Season 3" - you can again miss something.
You just need to create one distribution for everything and be able to select the .torrent file you need in one place and do not rummage through it all. You can put all sorts of tools nearby - what kind of subtitles are needed, what translation, what season, what quality of the picture, etc.
In order to be able to merge distributions you just need to separately sign them all - when there is some kind of distribution, and then more content appears, then it adds its file to the already existing one and it queues for moderation and this new content ( not the entire distribution) will be marked “not checked” (well, or as it will be better) until it is signed by the moderator and also everything as in paragraph 2. you just need to sign all of this separately and separately implement the functionality of clause 2. Looks nice.
Problem 4 - For a more or less fast download, a tracker is still required.
It's more complicated. It is necessary to increase the viability of the network - for this you need to implement the following algorithm to search for a new tracker.
To do this, we need the following functionality:
- Getting a suitable tracker for distribution from an almost arbitrary other tracker
- Redirect from one tracker to another
The first mechanism is needed in order to understand what to connect when the tracker falls, and the second if the current tracker is working, but does not want to handle this client.
With the second, everything is simple and understandable - just at announce, to return something like “redirect to ****” to it and that's it.
But for the first you need a special server setting - you need to create some network of servers (let it be a nexus), it is desirable that they are geographically distant. Each server from this nexus will have a prioritized list of this entire small network, which it will send to all its clients. If the tracker suddenly stops responding, clients will go down the list of these servers and connect somewhere. At this time, data is being shared inside the network and determining who will actually distribute the content now (this will happen very quickly - the network is small and just “agree”, and for implementation you can estimate some heuristics and come up with some simple and unambiguous rules to determine the successor of the deceased tracker), and then after you decided, then redirect all of this new tracker.
This should not cause any configuration problems - people already set up backups of their torrents, and here you just need to talk to people and almost don’t need to configure the server - they will do everything themselves.
Problem 5 - It’s not so easy to distribute on torrents
It is still important to remain anonymous and make this network more distributed. Here it is proposed to make it through the complication of the hierarchy, but the complication should be such that it only strengthens the network, rather than weakens it.
Our trackers are already merged into nexus and can form with each other some zone that can function as long as at least one member of the nexus is alive and all the content from the dead members of the nexus is stored accordingly. Yes, there is a problem with the load, but if there are 20 servers in a nexus, then there probably will never be such a problem.
If it turns out that the load is too large, then you can always raise the support server and redirect some of the clients, etc. to it.
In order to identify the server to participate in the nexus, digital signatures will be used, etc. This will protect the network from external influences. And the subsidiary subsidiary servers may have a different key and, accordingly, when creating a nexus, everyone exchanges a pair of keys - from the main server and from the subsidiary ones. Auxiliary can be as many as you want, but only one main one. This mechanism was created to simplify the connection and expansion of the network by one network participant - so that it would not be necessary for everyone to communicate and exchange keys. In fact, secondary servers are only caching proxies. But in general, their need is rather illusory.
Problem 6 - There is no possibility to automatically update torrents.
This is very easy to implement through the answer to announce, so we will not be distracted by it.
Problem 7 - Difficulty initiating distribution of content without using distribution on the tracker
To do this, you can simply implement for each user his own section, for distribution of which he can give links that can not be opened without, for example, a generated password. In principle, there is also nothing particularly difficult here - trackers are always there and will work. In principle, you can not even store everything on the servers, but you can make the tracker directly in the client - so that the user can make requests to this client via the XMPP server for these private distributions.
And these are all ideas?
No, not all, they are simply mass and over and above the goal (or over desire - as anyone) - is to create the perfect tool for the exchange of information. In addition to the above, there are a lot of introductions to all of this. For example, to create an anonymous semi-automatic method of merging information into the network (for a la wikiLeaks), to create opportunities for closed trackers (so that only some members of this super-network could participate in file sharing).
As for the interface, I want to make a single basic interface and I don’t want it to be a browser. The browser has too few features for real file sharing.
And in the pursuit of this - to have a certain hierarchy of Nexus, which would be clearly seen at the palm of your hand “the whole Internet”. For example, splitting on a territorial basis, on a direction (IT, music and tph), etc. And in the nexus themselves, too, create their own hierarchy, etc. I even think that it would be interesting to start with this. And m. so that the Nexus would not participate in the hierarchy, but the trackers themselves. It is likely that you will have to do something like DNS with the issuance of names in this hierarchy. But in general, there is already a lot to think about how to organize it, that there would be no cybersquatting, etc. Create the ability to create closed communities without your own tracker. But there is already a security issue - the server has all the information and in principle it can send everything to other network participants ... here you can use the “tracker in the client” mechanism with synchronization of network participants via an encrypted channel. In general, many, many good things.
Ok. And then what?
In general, I propose to put together a small team and start creating it all. If not us, then others. People are already fussing at the expense of distributed different things (for example, the diaspora). All will be useful.
- Programmers UI / Maker-ups - without you we will not make a good UI, as seen by usabilityists and designers!
- Server programmers - without you we can’t create anything reliable!
- Designers - without you, we can never create a beautiful system that would be pleasant for everyone to use!
- Usability / UI-design - without you we will not make a convenient and competent application!
Waiting for all in the comments.
Source: https://habr.com/ru/post/112682/
All Articles