Interplanetary File System IPFS

InterPlanetary File System is a new decentralized file sharing network. It also acts as a content delivery network .

The difference from other decentralized networks is that the unit itself is the unit transmitted in the network. A block can contain both part of a file and links to other blocks. A directed acyclic graph is built from the blocks from which a file or directory is subsequently collected.

Such a system allows a more flexible approach to the storage and transmission of data in the network. The disadvantage of this approach is that everything that is loaded into the network is cut into blocks and put in a separate directory on your disk. (Corrected: "no longer need to copy to the network" ) Search by file name or directory in IPFS is absent as well as in the BitTorrent network.

Multikhesh

Base58: QmcXx5mKDQAc7tCWLq84Hn7XFxWfBdZpvogJk3tNXQRFiv HEX: 12 20 D2E8A669F85E864B5C285CAB18DE72CF8927DC1643516373FEFF871A69F2A127 

The identifier in this decentralized network is the sha128 wrapped in a multi- hash from the block. A multi-hash consists of three parts: ID hash function, hash size in bytes, hash. To store the ID and size is used now one byte, but their values ​​cannot be greater than or equal to 128 until they are determined with the unsigned-varint format by which they are indicated. Multihesh in the links is encoded in Base58 bitcoin .

The multi-hash is used with a reserve for the future so that the network would have the opportunity to switch to new more stable hash functions.

Now the multihash is part of the content id .

Customer

 ipfs daemon 

In working condition for the IPFS network, there is a multiplatform client written by GO. You can download it and compile it yourself.

The ipfs daemon command starts the client that provides the web interface. You can enter it at the default address http://127.0.0.1 span001/webui

You can add a file and directory to the network using the command ipfs add [ ] or ipfs add -r [ ] . You can also add files via webui on page http://127.0.0.1 span001/webui#files .

After adding a file or directory, its multi-cache will be displayed. ipfs repo gc added in this way are automatically attached, that is, they will not be deleted if you issue the ipfs repo gc command. They also become available throughout the network through these multi-chests. If you open the web interface page http://127.0.0.1 span001/webui#objects you can see how the files are represented on the network by entering their multi-hash in the search bar. It shows the contents of the block.

Block

Above, I said that a separate unit on the network is a block. If you take from the page http://127.0.0.1 span001/webui#objects a multi-cache of a file block and add to it http://127.0.0.1:8080/ipfs/ then the network will give part of a file or directory corresponding to the block with this multi-cache.

For direct work with raw blocks, ipfs also has commands:

  ipfs block get <key> -  raw IPFS . ipfs block put <data> -    IPFS . ipfs block rm <hash>... -  IPFS (). ipfs block stat <key> -    raw IPFS . 

With the help of ipfs block get [] I found out what corresponds to the multi-cache from the test file I just uploaded with the command ipfs add [ ] using RHash and the Base58 online decoder . Files smaller than 262158 bytes are stored on the network in one block. The format of this block, I have not figured out the default is the data packaged in Protocol Buffers .

It is interesting that webui is stored in a decentralized network and part of the client’s code is also stored in it. The code is downloaded from the network automatically before compiling the client. Why this is done I do not know. This makes reading the source code a bit difficult.

Gateway

The IPFS network has access to a regular network. That is, through the website gateway.ipfs.io, you can access any file or directory available on the IPFS network. To do this, it is enough to know its multihash and add it to the address http://gateway.ipfs.io/ipfs/[] .

Some clients of decentralized networks have learned to pull data on http (s). The website gateway.ipfs.io can be used as a WebSeed. And if you have both IPFS and another WebSeed client installed at the same time, then the second will be able to use IPFS locally via http://127.0.0.1:8080/ipfs/[] .

In this article “ Why the Internet needs IPFS, it’s not too late ”, browsers suggest adding an extension that will change the address of ipfs.io to 127.0.0.1:8080 . This is necessary to view IPFS content through a local client in a browser.

Experiments with IPFS and BitTorrent

Experimenting with IPFS WebSeed and qBittorent, I noticed that the local IPFS is very slow in sending the data requested in a random order. Only when I turned on the serial download speed increased to 1MB / s. And that all distribution blocks are in the local storage. Also IPFS heavily loads the processor.

For BitTorrent IPFS, the client can serve as a good deduplicator. He can find copies of files on IPFS multihash directory or file. To do this, add it as a WebSeed to the torrent.

IPFS WebSeed

In order to distribute the contents of a directory using IPFS WebSeed, you need to transfer to another empty directory before uploading to IPFS and add it to the download ipfs add -w -r [ ] command ipfs add -w -r [ ] . The source directory will be accessed via the link http://gateway.ipfs.io/ipfs/[]/[ ] or http://127.0.0.1:8080/ipfs/[]/[ ] .

BitTorrent clients automatically add the full path to the file from the torrent to the WebSeed link in case it is a torrent directory. It’s enough to add a link like WebSeed http://gateway.ipfs.io/ipfs/[] or http://127.0.0.1:8080/ipfs/[] .

For a single-file torrent, simply adding the file with the ipfs add [ ] command ipfs add [ ] and WebSeed links to it will be of the same kind http://gateway.ipfs.io/ipfs/[] or http://127.0.0.1:8080/ipfs/[] .

To ensure that files do not occupy double space on the disk, you need to use the - --nocopy key. How to enable and use it in the article "no longer need to copy to the network . "

After that, the obtained links can be added to the created torrent file or magnet .

Magnet example:

 magnet:?xt=urn:btih:aaaaaaa...&ws=http://gateway.ipfs.io/ipfs/[] 

WebTorrent and IPFS WebSeed

WebTorrent can only work with the local IPFS gateway and only after configuration, as the gateway must give permission for WebTorrent to use the HTTP header Range.

To do this, open http://127.0.0.1 span001/webui#config

And in the line:

  "Access-Control-Allow-Headers": [ "X-Requested-With" ], 

add "Range":

  "Access-Control-Allow-Headers": [ "X-Requested-With", "Range" ], 

then save and restart IPFS

Interplanetary Name System

Also, each client has the opportunity to bind an object to the multi-cache of his public key. At the gateway.ipfs.io site, gateway.ipfs.io can access this object by adding the multi-key public key to the address http://gateway.ipfs.io/ipns/ . In order to bind or retrieve an object through a multi-key of a public key, there are commands:

  ipfs name publish <ipfs-path> -    IPNS. ipfs name resolve [<name>] -     IPNS . 

The combination of free DNS, http://gateway.ipfs.io/ipns/[ ] and IPFS client allows you to distribute a small static site from your home server even while you are behind NAT.

Interplanetary File System and Magnet

Now IPFS does not know how to work with magnet links, and yes, it is not described by the URN which contains the multi-cache used in the network.

I suggest this: urn: ipfs: [Base58 bitcoin encoded IPFS multihash]

In the magnet link, it will accordingly look like this:

 magnet:?xt=urn:ipfs:[Base58 bitcoin encoded IPFS multihash] 

Example:

 magnet:?xt=urn:ipfs:QmcXx5mKDQAc7tCWLq84Hn7XFxWfBdZpvogJk3tNXQRFiv 

By writing a small script, you can teach IPFS to work with magnets or other decentralized clients to use IPFS as WebSeed.

I think the use of magnet links will get rid of the ipfs.io site with which the same story can happen as with the shareaza domain . And more clearly identify IPFS multihash.

A bit off topic

I drew for magnets (actually wrote) a logo for free use. I want to present it so that it becomes a recognizable magnet-reference symbol and was used to designate it. Maybe then I can roll back the cancellation of my edits in the article on the magnet links and return the logo to the article.

he's on github

The colors for the magnet are taken from the customers' logos of the decentralized networks. I am not a designer. I tried to make drawing simple and compact. You are free to copy and modify it.

Sources

• IPFS
• IPFS command reference
• Multikhesh
• " Why does the Internet need IPFS before it's too late ?"
• " Work in IPFS with a human face "

Continuation

1. We publish the site in the interplanetary file system IPFS
   In this article I will tell you how to start a static site in it that will be accessible both directly and by IPNS. The site will have a normal domain name due to the use of DNS. The domain name can be used to access the site directly, through a global and local gateway.
   
2. We host a site in the interplanetary IPFS file system under Windows
   Some time has passed since the beginning of my experiments with hosting simple sites in IPFS. I launched my IPFS client under Windows and now I have something to add to the previous article.
3. No need to copy to the network
   The idea of ​​IPFS is good for everyone, but there was only one flaw with it. Data uploaded to the network were copied to the block storage by doubling the space they occupied. Moreover, the file was cut into blocks that are of little use for reuse.
   
   There is an experimental option --nocopy, which eliminates this drawback. In order to use it, it is necessary to fulfill several conditions.
4. Switch your site to localhost (local IPFS gateway)
   It makes little sense in IPFS, if you use it only as a free hosting for a site on the Internet. Therefore, we will learn how to download our site through the user's local IPFS gateway.
   
5. Localize the global gateway or sites in IPFS
   We will learn to switch to your local IPFS gateway sites that do not automatically do this themselves yet. Let's create them a shared SSL certificate using OpenSSL bundled with Stunnel.
   
6. Trivial hash (identity), DAG block and Protocol Buffers
   Recently, I added support for a trivial (identity) hash in IPFS. The trivial hash is the data itself. The data does not change at all and, accordingly, the size of the “hash” is equal to the size of the data. In my article I will tell about it and show how it can be used.