Web - P2P - Web

Good time of day, dear% username%.

Recently I read a habrastiya dedicated to a fairly common idea of ​​a distributed web. I myself have been busy for some time in the practical implementation of the project, the ideas of which are strikingly similar to the ideas of the author of the above article.

Under the cut, general thoughts, a review of the current state of the project, prospects and, of course, some traffic.
')
If interested - welcome under cat.

The idea of ​​a distributed Internet has long been wandering in the minds of all sorts of technical and subtechnical specialists. Someone suggests doing fully distributed torrents with blackjack auto- updates and without trackers, someone immediately takes on distributed OS (I don’t understand very well such a thing, but it sounds impressive), someone takes DNS over. There are already several projects that are suited to implementation from different angles, but the article is not about them.

General thoughts

Every time when it comes to the use of P2P technologies in relation to things other than torrent, the following advantages of using distributed networks are mentioned - anonymity, fault tolerance and performance.
Therefore, the system being developed must:

• provide the ability to fully anonymize the participant;
• ideally, do not have a single point of failure;
• work at the same speed or faster than traditional client-server systems;

Also, the possibility of sharing not only channels, but also other resources included in the network (disk, RAM, processor) is useful - this already applies to distributed computing.

Having a software platform that allows you to run part of the code on a peer-to-peer network, we can create any distributed services (network applications) for, for example, file sharing, generating web pages and sending messages. Such a distributed service program is essentially a set of functions, the execution of each of which can be delegated to a neighboring node, as well as an initialization function (entry point, specialized function that monitors the execution of the task).
Running applications over the network gives:

• the ability to use the resources of network participants - increasing the speed of processing a large number of requests;
• the absence of a single point of failure in the case when the initialization function is available on any node;
• complete anonymity when using mandatory encryption of connections between nodes;
• full backward compatibility with existing protocols and standards, since the output of the initializer function may be plain HTTP or another protocol;

Social part

Why does a distributed web project have a future in this form?

The platform, which has capabilities for scaling, is primarily interesting to webmasters and system administrators - they will be the first users. They will also be attracted by the high speed of calculations and the fact that users will not have to put all sorts of "improvers" for themselves. So, there will be no problems with the content. So, there will be users. Thus, we exclude the main problem of such distributed networks - unpopularity.

The absence of code on the client side means that the client can use the network services, but not enter it. To provide resources at the disposal of the network, it is enough to put a separate application, but this is a voluntary matter - we are not going to cram everybody “with an iron hand into a bright future.” Incentives to enter the network there. A good example are torrents. Ratings and statuses, the ability to use advanced features and other ways of motivation. More and more people will enter the network, ensuring its growth. With a sufficient network size, it will be possible to create truly 100% autonomous non-killed services. And this is the future, gentlemen.

For those who read the heading "social part", I thought about something else, I’ll inform you separately - yes, there you can make a distributed social network, read below.

Technique: what we are dealing with and how it works

When a request arrives (the request includes the network name, the name of the program being called and the initial data) at one of the network nodes, the node that received the request looks to see if it has the necessary code and whether it has the right to execute it. If there is no code, the node requests this code from its neighbors and receives a packet with a module containing the necessary (initialization) function. The initialization function, in turn, calls all the required subroutines using the internal API. In this case, the subroutines called through the API can actually be run on other nodes, just as the initialization function itself was launched. The determination of the most suitable node and the synchronization of the necessary data in this case falls on the API. The final results are returned to the parent function. At the same time, it is not possible to determine the address of the initially requesting device — the nodes know only the addresses of the location of the parent functions, and all connections between peers are encrypted (except in especially trusted cases, to improve performance).

Information in the network are presented in the form of data sources. If you need a file, string or number - you can request it in a convenient way for you from the corresponding function.
On top of this platform, you can already run the infrastructure - analog DNS in the form of a network application that accepts an address at the input, and gives you a list of IP (and / or a special network address). Yes, and the sites themselves become simply distributed applications generating and compiling HTML code and can be executed independently of any server.

Development

The main part of the project is written in the mighty and great language of Perl, but this does not mean that for professionals from other languages ​​(and not professionals - we are happy for each adequate person) there is no place and business.
Our project is just beginning its open life - there are tasks and there are many of them.

What is already done:

• Functional core for parallelization of subroutines;
• Feature level ACL;
• connection pooling;
• synchronization code nodes;
• synchronization of operational data;

That is, it is already realistic to launch a site that can quickly work on several nodes, and deliver the content via HTTP.

Common tasks to be solved:

• distributed data storage;
• a system for finding the necessary resources in the network by their identifier;
• analog DNS and plugins to support browsers;
• built-in protocol handlers;
• much more;

You can also read topics from the discussion of the project in the Google group (it is quite general - it includes topics not related to the project).

If you really want to twist in the hands

As I wrote a little higher, the site on which the development will be conducted and, accordingly, the code is published and the documentation is being prepared for launch (UPD: the site is launched). The code with the documentation is also being prepared for launch. In order to get a more or less high-quality start, it will all be published in about one and a half to two weeks (this is like going into an IPO, only more difficult). After the publication (and in the case of the habrasocommunity interest), a series of articles is being prepared on the use of the platform for programmers and administrators.

Perspectives

Sociality

Creating and using a social communication system, where everything is: my profile, personal page interface and all personal data are stored on my computer (with the full right to delete, if you suddenly decide) I see a much more interesting situation when all personal information is available to “someone else’s uncle” who also earns it. Therefore, I hope, to be a distributed social network - it is quite realistic to create it on the planned infrastructure. We write in perspective - for subsequent implementation.

Mail, telegraph, telephone

There also (in perspective) it is necessary to record a system of direct communication - text, document, voice, video and holographic (when they come up). You can try to build protection against spam at the level of architecture, but this is so ... Perspectives.

Monetization

I deliberately did not touch on anything in the article related to money and monetization of the project - this is a subject for a separate discussion (not even an article, but a discussion). Therefore, we also write monetization in perspectives.

Independence

The Internet lost its independence this year in 2005. Distributed Internet, part of which we are going to create here, of course, will not make you independent from providers (a vivid example of dependence), but, in the long term, will allow you to avoid the oppression of all kinds of RIAA and MPIA , but I am definitely not a friend of such organizations - I have my own thoughts on this topic with a volume of about the same article), to some extent protect against attacks on the DNS (I could not find a link to the article about the mass domain shutdown; if in comments someone will find - replace nude), and will help the emergence of new independent sources of information, which will make the world a little better that will give us new opportunities to receive it.
PS Thanks to everyone who helped me in publishing the article - without you it would not be here.

Materials:
http://groups.google.com/group/DarkMatterP2P/ is the place where all discussion is currently taking place on issues related to the topic, including those not related to the project.
http://habrahabr.ru/blogs/p2p/112491/ - quite an extensive article with a good selection of links.
http://habrahabr.ru/blogs/infosecurity/112682/ - another article on the topic, with an attempt to organize people and do something sane.