The decentralized data warehouse Ethereum Swarm

The Ethereum blockchain is interesting for its smart contracts, as well as the ability to create decentralized DAPS (DApp) applications. However, such an application needs a decentralized data warehouse.



Storing large amounts of data in a blockchain can cost a lot of money. Such decentralized repositories as the Ethereum Swarm come to the rescue (“swarm” is translated as “swarm”, “heap”). In short, Ethereum Swarm is a program code running on the Ethereum peer-to-peer network. It provides decentralized data storage on disk nodes, the owners of which give their resources for general use.

')

In this article, we will discuss how to install a local Ethereum Swarm node for the Ethereum private network in order to test technology and develop decentralized applications that store data in Ethereum Swarm.

Why DApp needs a decentralized data warehouse

The Ethereum blockchain (like other blockchains) is a distributed log, very well protected from fakes and attacks from intruders.



This protection is achieved, on the one hand, using hashing and cryptographic algorithms, algorithms for adding new blocks to the blockchain, and on the other, with a huge number of Ethereum nodes that provide data storage and transaction processing. These nodes are located around the world. No one has enough resources to “break” the blockchain, and this ensures the credibility of the information recorded in it.

“SkyNet is everywhere and there is no single center anywhere. There is nothing to disconnect. "(Terminator-3: Rise of the Machines)

But as I noted above, the blockchain is not suitable for storing large-scale data, such as documents, images, descriptions of goods, orders, etc. Theoretically, of course, you can write them in the blockchain, but the cost of recording will be too high.



Where to store data DApp large volume?



For example, you are going to use the blockchain to fix the copyright to the images, save forever information about an order made in an online store, or save scans of paper documents. Some other data storage is required here.



Of course, to store DApp data, you can rent a server, a place in the cloud storage, or use some other traditional storage. But servers, clouds and storage are controlled by companies or individuals. They can be blocked for any reason or attacked by intruders.



When using centralized storage, the main advantage of DApp is lost - resistance to attacks and actions of intruders. Moreover, your application ceases to be decentralized. This means that trust in DApp is also lost. After all, if you block or disrupt the operation of a centralized repository, access to some of the information recorded by DApp using the blockchain will be lost, and the whole point of using DApp will disappear.

How does Ethereum Swarm work

Imagine that tens of thousands of Ethereum Swarm network nodes are scattered around the world, which provide their resources for storing user-loaded data. It is assumed that the owners of the sites will receive a reward for the provision of resources, while the cost of placing the data will be lower than in traditional cloud storage.



When a user uploads a file to the Ethereum Swarm network, this file first gets to one of the nodes. Further, the file is replicated to other nodes in the network during the synchronization process. This uses the bzz protocol, running on top of the Ethereum network.



As long as at least one Ethereum Swarm node is running, the downloaded file remains available. This ensures reliable data storage, since It is almost impossible to disable or block a huge number of Ethereum Swarm nodes.



You can read more about all this on the Ethereum Swarm website.



The documentation notes that by now Ethereum Swarm has been implemented in version 0.2 as proof of concept (POC, proof of concept). In this version, the safety of the downloaded data is not guaranteed. Stable release is expected in the second quarter of 2018, so the wait is long. At the same time, it is planned to create a reward system for providing resources for the Ethereum Swarm network.



However, we will not wait for anything, but we will start testing Ethereum Swarm right now. We will create our own Ethereum Swarm node in our private Ethereum network.

Installing Ethereum Swarm and Geth on Debian and Ubuntu

I will talk about installing Ethereum Swarm on Debian and Ubuntu servers. In general, you need to act on this instruction , but there are some nuances.

Go installation

Before starting work, it is very important to install Go of the new version, not lower than 1.9.2. Debian and Ubuntu repositories may have older versions of Go, so install from source.



Download and unpack the sources under an unprivileged user:

$ curl -O https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz $ sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz 

Create a go directory for the user and set the environment variables:

 $ mkdir -p ~/go; echo "export GOPATH=$HOME/go" >> ~/.bashrc $ echo "export PATH=$PATH:$HOME/go/bin:/usr/local/go/bin" >> ~/.bashrc $ source ~/.bashrc 

Check that the environment variables are set:

 $ printenv | grep go PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/frolov/go/bin:/usr/local/go/bin GOPATH=/home/frolov/go 

On the Debian operating system, use the .profile file instead of the .bashrc file. You can simply copy:

 cp .bashrc .profile 

Check the go version:

 $ go version go version go1.9.2 linux/amd64 

The go version must be at least 1.9.2.



If the old version of go from the OS repository was previously installed, delete it as follows:

 $ sudo apt-get remove golang-go $ sudo apt-get remove --auto-remove golang-go 

Install Geth and Ethereum Swarm

Download the source code from the repository:

 $ mkdir -p $GOPATH/src/github.com/ethereum $ cd $GOPATH/src/github.com/ethereum $ git clone https://github.com/ethereum/go-ethereum $ cd go-ethereum $ git checkout master $ go get github.com/ethereum/go-ethereum 

We start compilation of the geth client and the swarm daemon:

 $ go install -v ./cmd/geth $ go install -v ./cmd/swarm 

Check the version of the installed geth and swarm:

 $ geth version Geth Version: 1.8.0-unstable Architecture: amd64 Protocol Versions: [63 62] Network Id: 1 Go Version: go1.9.2 Operating System: linux GOPATH=/home/frolov/go GOROOT=/usr/local/go $ swarm version Swarm Version: 1.8.0-unstable Network Id: 0 Go Version: go1.9.2 OS: linux GOPATH=/home/frolov/go GOROOT=/usr/local/go 

Preparing a private blockchain to launch Ethereum Swarm

First of all, create the file genesis.json in the user's home directory:

 { "config": { "chainId": 1907, "homesteadBlock": 0, "eip155Block": 0, "eip158Block": 0 }, "difficulty": "40", "gasLimit": "5100000", "alloc": {} } 

Next, create a node1 subdirectory in your home directory:

 $ mkdir node1 

The node can be initialized using the init_node.sh batch file:

 geth --datadir node1 account new geth --datadir node1 init genesis.json 

When you run this file, an account will be created and a password will be requested, which you need to keep in a safe place.



Create a start_node.sh file to start the node:

 geth --datadir node1 --nodiscover --mine --minerthreads 1 --maxpeers 0 --verbosity 3 --networkid 98765 --rpc --rpcapi="db,eth,net,web3,personal,web3" console 

Run this file and wait for the DAG generation to complete.



Using the attach_node.sh file, you can open the geth console and connect to a private node:

 geth --datadir node1 --networkid 98765 attach ipc://home/frolov/node1/geth.ipc 

Starting the swarm daemon

Here you will need the address of the account created during the node initialization stage. If you did not save it, do not worry. Just log into the geth console, opened with the attach_node.sh script, and issue the following command there:

 > web3.eth.accounts ["0xcd9fcb450c858d1a7678a2bccf36ea5decd2b09b"] 

The command will show the address of the created accounts. Immediately after initialization, there will be only one address.



To start the Ethereum Swarm daemon in singleton mode, prepare the swarm_start.sh command file:

 swarm --bzzaccount cd9fcb450c858d1a7678a2bccf36ea5decd2b09b --datadir "/home/ethertest/data/node1" --maxpeers 0 -nodiscover --verbosity 4 --ens-api /home/ethertest/data/node1/geth.ipc 

Specify the address of the account created on your site, without the "0x".



When you start the daemon, you will need to enter the password for the account you created earlier:

 $ sh swarm_start.sh Unlocking swarm account 0xCD9Fcb450C858D1A7678a2bCCf36EA5decd2B09B [1/3] Passphrase: 

Download the file to Ethereum Swarm

The easiest way to download a file is using the swarm command with the up parameter. Additionally, you need to specify the path to the downloaded file:

 $ swarm up start_node.sh f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f 

This command will return the hash of the downloaded file. The hash can be used to read the file. You can do this with the wget or curl commands:

 $ wget http://localhost:8500/bzz:/f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f/ $ curl http://localhost:8500/bzz:/f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f/ 

The wget command allows you to save the contents of a file on a local disk. Use the -O option to set the file name. The curl command will output the contents of the file to the console, so in this form it should not be used to view the contents of binary files. A slash is required at the end of the URL, otherwise a redirect will occur.



When a file is loaded into Ethereum Swarm in the manner described above, a so-called manifest is created and saved for it. This is a header that describes the content available in the repository with a given identifier.



Below we downloaded the Net-Ethereum-0.28.tar.gz file using the simple swarm up command:

 $ swarm up Net-Ethereum-0.28.tar.gz 8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f 

Now, having specified the bzz-list protocol, we can view the manifest:

 $ curl http://localhost:8500/bzz-list:/8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f/ {"entries":[{"hash":"543ee6e744f93de76ac132b8ab71982e32beaf90d1005e771dde003b2a4a54c3","path":"/","contentType":"application/gzip","mode":420,"size":12403,"mod_time":"2018-01-13T14:57:54+03:00"}]} 

The manifest will be shown in JSON format.



The manifest stores the file (file name), its size, type (Content Type), modification date and time, as well as the hash of the file.



To extract the contents of the file by its identifier and save as t.tar.gz, do this:

 $ wget -O t.tar.gz http://localhost:8500/bzz:/8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f/ 

Downloading directories with subdirectories

To recursively load a directory with its contents in the Ethereum Swarm storage, specify the --recursive parameter:

 $ swarm --recursive up Net-Ethereum/ 4fb1f2270381c022461037151f70ce081082f0ae1a2a23d8c7ea602da69b4115 

The manifest will show information about all files in the loaded subdirectory:

Other features of Ethereum Swarm

In this article, I did not talk about all the possibilities of Ethereum Swarm, limiting myself only to the most necessary.



In addition to downloading individual files and directory contents, you can add data to the manifest. This creates a new data item with an added manifest and with the added files.



It is possible to download files without a manifest, as well as read them in the so-called "raw" (raw) form.



When reading a file, you can specify its name (in whole or in part). This will involve the name service ENS (Ethereum Name Service). Such an opportunity will allow you to get only some files from a block united by a common manifest.



I also did not describe the SWarm Accounting Protocol (SWAP) and some other features. The Ethereum Swarm project is actively developing, there will definitely appear something interesting for developers of decentralized applications.

Perl Net :: Ethereum :: Swarm Module

In order to work with the Ethereum Swarm decentralized data warehouse in systems written in the Perl language, I developed and posted the Net :: Ethereum :: Swarm module on CPAN.



With this module, you can load text and binary files into Ethereum Swarm, get a manifest for the downloaded data, and download files from Ethereum Swarm by their identifier.



The Net :: Ethereum :: Swarm module works with the Ethereum Swarm node using HTTP GET and POST requests. The use of requests is described in The HTTP API documentation section.



Interaction via HTTP requests can be easily implemented in almost any programming language.

My other articles about blockchain and Ethereum smart contracts

• Experiments with Solidity contracts on the Rinkeby test network of the Ethereum blockchain
• Net :: Ethereum module for working with Solidity contracts