The birth of magic from the spirit of the torrent protocol (scheme of the organization of a free uncensored indestructible information network)

Formulation of the problem

As is known, the Internet was created as an information transmission network capable of operating in a nuclear war. The communication channel between two points everywhere and nowhere, removing one of the nodes of the network will not prevent the others from exchanging packets. However, this distributed data transmission network is not a distributed storage network - in principle, data is stored in a specific place and in some cases, the vulnerability of end nodes is very ... disturbing.

It is probably time to overcome this limitation. The technologies that allow to do this have already been invented and widely known, it remains only to find the right way to put them together.

The option of a data network with a distributed source is a familiar torrent protocol for all of us. It is clear, however, that it was designed for the narrow purpose of file transfer and does not allow for the construction of a complete and self-sufficient system.
')
I propose a scheme for creating such a system, starting from the principle of the functioning of torrents.

So, we will solve the following problem:
creation of a decentralized distributed network of storage and transmission of information,
with anonymous but identifiable participants,
acting on an equal footing and not needing any kind of single governing authority,
which provides storage and distribution of any information as long as there are participants in the system who are interested in distributing it.

The key to its solution is the organization of a distributed and decentralized addressing system.

Distributed Decentralized Addressing System

You can create it in the simplest way. We call an atom an elementary unit of information that can be addressed in the system. (An atom will be the equivalent of a file or Internet page). In order to create an atom, the participant of the system must sign it with his electronic signature (according to the public and private key scheme).

The resource in the system (similar to the Internet site) will be identified by the public key. It is not necessary to enter any other address or identifier operating at the system protocol level. This will allow the addressing system to be completely decentralized and not in need of organized control.

An atom is identified by the address of the resource plus the crypto-resistant hash function of the atom itself. An atom must include a block of meta-information (with the address of the atom and an indication of the type of atom, that is, the protocol by which the atom should be read) and the actual content, which is addressed and transmitted as a whole at the level of the main system protocol.

Let's call the owner of the key (resource) the publisher, and the resource - the publisher. It turned out - in the first approximation, the finished version of the addressing system we need.

However, in order to work comfortably with publishers and their atoms, we need another entity — the file (also an atom) of the meta-information of the resource. Let's call it a publication. The publication specifies the grouping of atoms and the bonds of atoms between themselves. In particular, the publication will allow the publisher, as necessary (creating a new publication) to introduce new versions of atoms (and the reader to find the current version at the old one). It is assumed that for publication files it will be possible to use several formats (at the choice of the publisher) - an obvious option seems to be the organization of the folder structure, it is possible to use methods for querying non-relational database data.

The publication provides the second, indirect type of addressing of the atom - through the address of the publication and the internal address, the construction method of which is specified by the format of the publication.

The publication format should also provide for the possibility of an additional signature of the publication of one publisher - another publisher, so that both (or more) signatures are available through the meta-information of the publication atom and are taken into account at the level of the main protocol of the system.

Now our decentralized distributed addressing system is almost complete. You can call it SRDS (self-distributed distributed addressing system). It differs in that it addresses (in a special way prepared) blocks of information, which are determined by their content - and nothing else. The content of each atom clearly corresponds to its address and can have any number of copies. The identity of each atom to the owner of a certain key (the publisher) is also uniquely determined by its content.

Magic Torrent Information Distribution Protocol

We return to the world of the usual Internet. It remains to solve a very simple task - to organize access to our atoms, which can be anywhere, but uniquely identified by their address. Let's use for this the idea of ​​the torrent protocol: the publication file is transmitted to the distribution organizer (let's call it the node); Atomic owners who want to join the distribution (sid) are registered from him - and those who want to obtain an atom (litchi) receive from the node the addresses of the sidics who have this atom and request the atom directly from one of them. (If necessary, atoms can be divided into blocks and requested block by block; such a breakdown must be specified through the publication file when it is created).

What addresses are we talking about here? I think it is desirable to implement multi-protocol work — that is, any participant can be present in the system under the address he is most comfortable with using either a regular TCP numeric address or, if desired, an address on the tor network, i2p network, and so on. The first implemented exchange protocol, IMHO, the most reasonable way to do https.

To get node addresses, you need to organize another, higher level - let's call it supernody. Supernodog in exactly the same way registers the nodes corresponding to our publishers and publication files. (Supernode is enough to store the publisher's SRDS address, the hash function of the publication file and the addresses of the nodes distributing it).

For the effectiveness of the work of the node (distributing the same resource) and supernods (preferably all existing ones) should be combined into a ring - each participant of which copies to itself all the information that any other one has and exchanges it in a permanent mode.

A client who wants to receive system information (we will call this client a visitor), say, by clicking on the SRDS address link — must directly get the full address (resource-publication-atom) from this link, go to the supernodus, get the node addresses that distribute the necessary resource , contact any of them, get the address of the side that gives out the required atom (or, if desired, the current version of this atom).

Let's call such a system for organizing access to SRDS resources - the Magic Torrent protocol.



Such an organization has a very high stability - the withdrawal of any participant from it has virtually no effect on the functioning of the system. Moreover, even in the case of complete destruction of the structure - removal of all nodes and supernods - it can be easily recreated by the remaining siders, who will take on the functions of nodes and supernods in a proactive manner.

Unfortunately, there are reasons why our system cannot be completely anarchic and decentralized. The supernodus ring is a critical resource for it, which should be controlled and moderated in some way - if either the number of participants or the number of publications expands in it, this will negatively affect the efficiency of the system. I believe that it should be formed according to the principle of a closed club, and publications should be accepted with a certifying signature (on top of the publisher's signature) authorized by the moderator club members. Naturally, after some time there will be several such clubs, and, accordingly, several rings - but natural selection will obviously leave only a small number of widely known and popular ones that can already agree on the principles of integration. Moreover, the strict selection of publications is most likely not required - it is enough to limit the reception of maliciously generated fake resources.

Since the system is designed for the free exchange of information, it is desirable to provide mechanisms that protect the information, including from arbitrarily deleting it by the publishers themselves (or by the attackers who seized their keys). When designing the work protocol of the nodes and supernods, it should be understood that the creation and registration of a new publication by the publisher (via the super-inode) is not a directive order for all the nodes to switch to the distribution of this publication, but a proposal that may be accepted by some, some may and some will wish to distribute several publications of one resource at a time. It is also reasonable to provide flags that will inform visitors about such versions of publications and the purpose of their preservation.

The default mode should rather focus on saving the notes of all publications and all atoms of the resource (including outdated editions) or only on excluding those publications that do not contain information that is not available in later ones.

Benefits and Applications

The advantages and disadvantages of the organization described are, in principle, obvious. It provides easy creation of resources, the possibility of free storage of information on a voluntary basis, easy and natural scaling, as well as duplication, if necessary, if it becomes necessary to reproduce the content of an information system in a partially isolated technical (alternative protocols, alternative networks), physical ( let's say, on other planets of the solar system) or legal reasons a segment of the Internet. Magic Torrent protocol can be used for backup channels, resistant to ddos-attacks. Addressing SRDS is ideal for implementing the long-standing idea of ​​distributed blogs, and generally for organizing network projects, the fate of which is undesirable to trust the arbitrariness of a proprietary service provider (say, freelance exchanges). But, obviously, the first decision is to use it to create network libraries.

The proposed system is not an alternative to anonymous networks - it can complement them, if necessary, use their capabilities, and they can provide a natural bridge to create a common space with the traditional Internet.

The main problem is the need to implement from scratch all the proposed protocols and programs - clients and servers of seeders, nodes and supernods. A separate big topic is the organization of metafiles and atoms, navigation and viewing methods convenient for working in Magic Torrent. The “binding” technology in the usual web is html - but for our needs it will have to be limited, modified or replaced.

Disclaimer:
1) As far as I know, the system proposed by me is fundamentally different from the set of existing or developed ones, but I do not pretend to be fully aware and I cannot give guarantees. If someone pokes a finger and says - “that's the same thing” - I think you need to pay everyone's attention to such a project, because it deserves greater fame.
2) Neither the intention, nor the ability, nor the qualifications to implement the ideas presented I do not have.

At the same time, the scheme invented by me seems extremely powerful, simple and promising. Its implementation can radically expand our capabilities and - after some time - noticeably change lives.

EDS