Why the Internet needs IPFS before it's too late

IPFS has not quite become a well-known technology, even in Silicon Valley to many it is not yet known, but news of it quickly diverges from word of mouth in the open source community . And many are excited about the potential of IPFS to improve file transfer and speed up streaming over the Internet.

From my personal point of view, however, IPFS is in fact much more important than these features. IPFS saves sites from having to use a central source server and therefore, this is probably our best chance to completely change the architecture of the Internet before it falls apart from internal contradictions.

How and why is it? To answer this question will have to go into details.

Why our World Wide Web is slow, fragile and forgetful

IPFS is a new p2p-hypermedia protocol, the purpose of which is to complement (or even replace) the hypertext transfer protocol (HTTP), which now dominates WWW. This is what the HTTP problem is: today, when you visit a site, your browser must communicate directly with the servers on which this site works, even when these servers are far away, and the transfer process requires considerable bandwidth.
')
Providers incur costs because peer-to-peer agreements are concluded between networks, so sending data from the network to the network costs the provider money and consumes bandwidth. Worse, the HTTP file is downloaded from just one computer instead of simultaneously receiving parts of a file from different computers.

That is why we are stuck in this state of affairs: a slow and expensive Internet, the price of which is still increasing due to the predation of the “last mile” providers (at least in the USA), and with the accelerating growth of connection requests from mobile devices. It is not just slow and expensive, but unreliable. If at least one link of the HTTP transfer is terminated for any reason, the transmission will also terminate. (And when a web page or media file loads for a long time, it is most likely due to one of the intercomputer connections.)

Redoing the Internet through IPFS

IPFS (InterPlanetary File System, i.e., interplanetary file system: this name is a tribute to the ideas of JK R. Liclayder on the "intergalactic" Internet) was invented by Juan Benet, who as a teenager came to the USA from Mexico, at Stanford received a degree in computer science, founded the company acquired by “Yahoo!” in 2013 , and last year through Y Combinator founded Protocol Labs, which are now engaged in the IPFS project and modestly intend to change the protocols that have been our everyday life for the last 20 years .

IPFS is a P2P-distributed file system that aims to connect all computing devices with one common file system; as such, it is able to exceed the capabilities of HTTP in several ways. Of these, two (as Joan told me in a recent conversation) are key:

“We use content addressing, so the content becomes independent of the source servers and can be stored separately and for a long time. This means that the content can be stored and distributed very close to the consumer - perhaps even a computer from the same room. Content addressing also allows you to check the data received from other (not trusted) hosts instead of the original source. And when the content gets to the user's device, it can be cached there as long as possible. ”

IPFS also eliminates some security issues that plague our HTTP-based Internet: content-based addressing (and subsequent verification) protects IPFS-based sites, IPFS distributions make DDoS attacks impossible. Helping to mitigate the damage from the closure of sites by their owners, IPFS also caches content that is important to society and can securely save it.

The main advantage of IPFS is the decentralized distribution of content, so that content from the Internet can be accessed under the circumstances of irregular access to the Internet and even taken from the offline cache. “We are creating websites and web applications that do not have a central source server, ” Juan explained. “ They can be distributed just like the Bitcoin network is distributed .” This is just what HTTP cannot achieve, so IPFS will benefit networks that do not have first-class connections (for example, in developing countries or in rural areas).

The emergence of the IPFS alpha version last February has already led to many experiments among early adopters . For example, on September 8, Neocities hosting became the first major site to use IPFS for storing content , that is, following the appeal of the Internet Archive about the need for distributed WWW . Currently, we all are undergoing an incessant loss of more and more new sites, year after year, abandoned and closed by their creators, and in this growing crisis of our collective Internet memory, even a small step towards the permanent Web is important for us.

And sites belonging to large corporations, will follow the example of Neocities, will the implementation of the not yet fully tested protocol begin - especially when even the mere mention of P2P can terrify them? And here I come to the last paragraph of the article.

Why IPFS is very important for the future of Internet businesses.

As I explain in my new book (the publication of which is still to come), we are rapidly approaching the point at which the cost of content delivery will outstrip utility - and financial gain. The largest companies of the Internet are trying their best to keep up with our demand for content, whole armies of engineers are engaged in this task alone in Akamai, Google and Amazon.

Soon their situation will deteriorate. Thanks to the rapid sale of cheap smartphones, entire continents of new consumers will come online in the next decade. The Internet of Things also promises to aggravate this problem, because billions of devices with their own inquiries will fall into the rapidly dwindling possibilities of our communication channels.

We need to limit the effect that I call micro-singularity to the extreme: the rapid "viral" drawing close attention of billions of Internet users to the same event threatens to strangle the entire system. (The interruption of the network’s operation theoretically threatens life as well, when a catastrophe, accident or other phenomenon of the same kind becomes micro-singular.)

Netflix has recently begun researching large-scale P2P streaming technology , and this is the primary and reassuring sign that companies of this size are looking for more reasonable methods for delivering content. And Netflix, and YouTube, and all the major traffic providers for us could flourish on the basis of the Internet, modified by the arrival of IPFS and an impressive decrease in the cost and time of content delivery.

By improving connectivity, IPFS will help the Internet become the system that we have always strived to see it in our idealizations, but which with the current protocols it could not: truly capable of connecting everyone in the world (even being offline) to a permanent, but who we are and what we strive for.

---

Afterword from the translator

Habrahabr is not even Silicon Valley, so even here, as can be expected, by the beginning of October 2015, not all readers had a sufficient idea of ​​IPFS for themselves. Consequently, after finishing a translation of an article from TechCrunch, I will also have to attach a definite afterword to it; and enclose it, putting it in the form of questions and answers.

- What is IPFS?

- Distributed file system with content addressing.

- In what sense is the IPFS system distributed?

- In the sense of P2P file sharing. After downloading the file, the user becomes its source for other downloads.

- In what sense is the IPFS system a file system?

- In direct: if your operating system is Unix-like, then using FUSE, you can connect to IPFS in such a way that you can see the IPFS files at “/ ipfs / hash file” in your file system.

- In what sense is the IPFS system content-addressable?

- File hash depends on file content only. If the file has a different name or lies in another subdirectory, then it is still the same file. (This differs favorably, for example, from BTIH’s Bittorrent’s hash, which changes depending on the file name and location.) If the same file is distributed under a different name or as part of another subdirectory, the distributions will automatically merge, no doubling of effort will be required, disk space.

- Does the file system IPFS subdirectories?

- There is; these are lists of names (and hashes) of files and subdirectories. For example, to open the address “/ ipfs / hash_directory / file_name”, the directory is first downloaded (by its known hash), then the name of the file in the directory reveals the hash with which IPFS will download the file. Each subdirectory name adds one step to this process.

- How do IPFS nodes know which of them has the desired file?

- Used DHT (like Kademlia , but with greater resistance to the Sibyl attack ). There is no central server that could be captured, disconnected, subtracted a name (like torrent trackers are captured and disconnected , for example).

- How can the site of the current World Wide Web refer to a file lying in an IPFS distributed file system?

- The author of the site sets itself IPFS and puts the file in IPFS. He then writes “ https://ipfs.io ” in front of the IPFS file path and gets, for example, “ https://ipfs.io/ipfs/QmcXx5mKDQAc7tCWLq84Hn7XFxWfBdZpvogJk3tNXQRFiv ”. The site “ https://ipfs.io ” works as a gate from IPFS to the WWW, so that readers who have not installed IPFS support can get a file from this site.

- And readers who have IPFS support installed?

- And they have a personal gate from IPFS. It remains only to put such an add-on for Firefox or an extension for Chrome , which will “127.0.0.1:8080” automatically substitute for “ipfs.io” in such URLs.

- How does IPFS save in the event of a network disruption?

- If the illustrations (and other files) are not published to the IPFS address from the very beginning, but in IPFS, then even if the site falls, the reader can still receive files at a known address, but not from the IPFS site cache, but from the cache neighbor. (And the neighbor will not be able to replace the file, because the file will be checked for hash.)

- How does IPFS help websites save traffic?

- If illustrations (and other files) are not published to the IPFS address from the very beginning, but in IPFS, then even if micro-singularity happens and a billion viewers very quickly come to the site, most of them will not be able to knock on IPFS anymore , and in a cache to the neighbors in a billion, who had a little earlier. So the site does not lie down. (Of course, for this, first the fashion should spread to the IPFS among the audience, but instead of that, instead of the site, the gate “ https://ipfs.io ” will lie down, for example.)

- How does IPFS help sites in the event of a disk crash?

- If you can publish illustrations and other static files from the very beginning not at the site address, but into IPFS, then you can download the missing pictures (and other files) via IPFS from the site caches if anything happens.

- How does IPFS help readers in the case of state censorship?

- If illustrations and other static files are not published to the IPFS address from the very beginning, but in IPFS, then it will be possible to forbid illustrations (for example, illustrations for the い 霊 使 い 舞 ) and other prohibited files to receive over IPFS bypassing the blocking ( from the readers caches of the site, which managed to get the files before the censorship took effect).
- How does IPFS help sites save disk space?

- If the illustrations from the very beginning are not published at the site address, but in IPFS, then you can erase them from the IPFS cache after some reasonable time, freeing up space and hoping that the illustrations will continue to be accessible to readers who will simply receive them from ipfs cache each other.

- That is, this is a real alternative for those image hosting and for imageboard, which are now working hard and after a certain time simply erase files irretrievably?

- Absolutely. (I suggested this alternative on Yichane, for example.)

- Is there an example of such a hosting of images right now that adds images to IPFS?

- Yes, ipfs.pics .

- Is there an example of hosting such arbitrary files right now that adds them to IPFS?

- Yes, http://ipfs.stadja.net/upload/ Alas, no: closed .

- Does content addressing mean that only static (immutable) content can be stored in IPFS?

- In general, yes. However, there is an IPNS name system for sites, which allows each IPNS name to be associated with its own IPFS directory hash, and then change this correspondence as needed. That is, put on such a site some static content, but new.

- Maybe invent a new URL scheme for IPFS? This will allow them to be written shorter than the long “ https://ipfs.io ” address in front of the hash.

- Two weeks discussed and decided to stop on the scheme "fs:", preceding such a path in the file system, which begins with "/ ipfs /" or "/ ipns /".

- Why didn't you make two separate schemes “ipfs:” and “ipns:”?

- For the convenience of users of Unix-like systems, which now only need to erase the three characters "fs:" to get the path. (Otherwise you can still mess around.)

- Will the “fs:” addresses be maintained in the hypertext Fidonet?

- There will be; three lines in ECMAScript 6 are enough for this. Here is an example of Fidonet blogging, first provided with an illustration from IPFS, and then translated via RSS to LiveJournal.

- Is it possible to add support for opening “fs:” addresses in the GoldED-NSF feedmail editor ?

Can:

GroupURL IPFS URLEngine PCRE URLScheme /ipfs/[1-9A-HJ-NP-Za-km-z]+ URLHandler start "" "https://ipfs.io@url" EndGroupURL 

- Does IPFS work under Windows?

The current alpha version under Windows works pretty badly , but the previous one has stopped running at all , so we can hope for further improvement in the future, and I hope.