How to make your blockchain. Part 1 - Build, Store, Sync, Display, Mining, and Proof

Good all! We slowly began to explore a new direction for us to learn - blockchains and found something that turned out to be interesting as part of our Python course, including. What, in fact, want to share with you.


I can find out when I got my first Bitcoin, from the history of the wallet in my Coinbase account - an incoming transaction in 2012 as a gift for registration. Bitcoin at that time cost about $ 6.50. If I had saved those 0.1 BTC, at the time of writing this article it would have cost more than $ 500. If anyone is interested, I sold them when Bitcoin cost $ 2000. So I only got $ 200 instead of the current $ 550. Not worth the rush.

I knew about the existence of Bitcoin, but I was not particularly interested. I have seen the ups and downs of the $ / BTC rate. I saw people say that the future was with him, and I saw articles about his complete meaninglessness. But I had no personal opinion - I just watched from the side.
In the same way, I almost did not follow the blockchains. But lately, my father mentioned several times that on CNBC and Bloomberg, which he watches in the morning, they often talk about blockchains, and he has no idea what it is.
')
And then I suddenly realized that I needed to sort out this topic a little deeper. And he began with a "study" - read a huge number of articles on the Internet, explaining their essence. Some were good, some were bad, some were deep, and some were very superficial.

Reading was not enough, and if there is one thing that I know for sure, it is that reading does not explain even a hundredth of what programming explains. And so I realized that it was worth writing your own local blockchain.

It should be borne in mind that there is a big difference between the basic blockchain, which I describe and the “professional” blockchain. This chain will not create a cryptocurrency. Blockchains do not require the production of coins that can be sold and exchanged for physical money.
Blockchains are used to store and confirm information. Coins encourage sites to participate in validation, but their presence is not required.

I write a post for several reasons: 1) So that people who have read it can learn more about blockchains; 2) So that I can understand more by explaining the code, rather than simply writing it.
In this post, I will show you how to store the blockchain data and generate the initial block, synchronize the node with the local blockchain data, display the blockchain (which will later be used to synchronize with other nodes), and then mine and create valid new blocks. In the first post there will be no other nodes. No wallets, peers, important data. We will talk about them later.

In a nutshell

If you do not want to go into details and read the code, or if you stumbled upon this post, counting on an article that would explain the blockchains in an understandable language, I will try to briefly summarize how they work.

At the highest level, the blockchain is a database where everyone who participates in the blockchain can store, view, confirm and never delete data.

At a lower level, the data in these blocks can be anything as long as a particular blockchain allows. For example, the data in the Bitcoin blockchain is exclusively Bitcoin transactions between accounts. The Ethereum blockchain allows both similar Ether transactions and transactions used to run code.

Before a block is created and merged into a blockchain, it is confirmed by the majority of people working on the blockchain — they are called nodes. A real blockchain is a chain consisting of a huge number of blocks, confirmed by the majority of nodes. Thus, if the node tries to change the data of the previous block, the new blocks will not be valid, and the nodes will not trust the data from the incorrect block.

Don't worry if this is confusing. It took me a while to get into it myself, and even more time to write such a post, so that even my sister (who knows nothing about the blockchains) could understand.

If you want to study the code, look at the part 1 branch on Github. Feel free to send me any questions, comments, edits and praises (if you are in the mood to do something particularly good), or just tweet.

Step 1 - Classes and Files

The first step is to write a class that handles blocks when starting nodes. I will call this class Block. Honestly, there is not much to do. In the function __init__, we will believe that all the necessary information is already presented in the dictionary. For a production blockchain, this is not the wisest decision, but it is suitable as an example, because only I am writing code. I will also write a method that packs important block information into a dictionary, and then I’m getting a more convenient way to display block information when it is printed to a terminal.

class Block(object): def __init__(self, dictionary): ''' We're looking for index, timestamp, data, prev_hash, nonce ''' for k, v in dictionary.items(): setattr(self, k, v) if not hasattr(self, 'hash'): #in creating the first block, needs to be removed in future self.hash = self.create_self_hash() def __dict__(self): info = {} info['index'] = str(self.index) info['timestamp'] = str(self.timestamp) info['prev_hash'] = str(self.prev_hash) info['hash'] = str(self.hash) info['data'] = str(self.data) return info def __str__(self): return "Block<prev_hash: %s,hash: %s>" % (self.prev_hash, self.hash) 

To create the first block, run this simple code:

 def create_first_block(): # index zero and arbitrary previous hash block_data = {} block_data['index'] = 0 block_data['timestamp'] = date.datetime.now() block_data['data'] = 'First block data' block_data['prev_hash'] = None block = Block(block_data) return block 

Fine. The last question in this part is where to store data in the file system. This is necessary if we do not want to lose the local block data when the node is disconnected.
I will name the 'chaindata' folder, to some extent imitating the Etherium Mist folder scheme. Each block is now assigned a separate file, named for its index. You need to make sure that the file names contain a sufficient number of zeros at the beginning, so that the blocks are listed in order.

Given the code above, you need to write the following to create the first block:

 #check if chaindata folder exists. chaindata_dir = 'chaindata' if not os.path.exists(chaindata_dir): #make chaindata dir os.mkdir(chaindata_dir) #check if dir is empty from just creation, or empty before if os.listdir(chaindata_dir) == []: #create first block first_block = create_first_block() first_block.self_save() 

Step 2 - Synchronize the blockchain locally

Before you start mining, interpreting data, or sending / creating new data for a chain, you need to synchronize a node. In our case, there are no other nodes, so I’m talking only about reading blocks from local files. In the future, part of the synchronization will not only be reading from files, but also communicating with peers to collect blocks that were generated before your node was launched.

 def sync(): node_blocks = [] #We're assuming that the folder and at least initial block exists chaindata_dir = 'chaindata' if os.path.exists(chaindata_dir): for filename in os.listdir(chaindata_dir): if filename.endswith('.json'): #.DS_Store sometimes screws things up filepath = '%s/%s' % (chaindata_dir, filename) with open(filepath, 'r') as block_file: block_info = json.load(block_file) block_object = Block(block_info) #since we can init a Block object with just a dict node_blocks.append(block_object) return node_blocks 

While simple and beautiful. Reading lines from files loading them into data structures does not require overly complex code. While it works. But in future posts, where I will write about the communication capabilities of different nodes, this sync function will become much more difficult.

Step 3 - Displaying the blockchain

Now our blockchain is in memory, and therefore I want to display the chain in the browser. To do this right now, there are two reasons. First, you need to confirm in the browser that the changes have occurred. Secondly, I will use the browser in the future to view and perform any operations related to the blockchain. For example, sending transactions or managing a wallet.

For this, I use Flask - it has a low threshold of entry, and I decided that it is suitable for our purposes.

Below is the code to display json blockchain. I will ignore imports to save space.

 node = Flask(__name__) node_blocks = sync.sync() #inital blocks that are synced @node.route('/blockchain.json', methods=['GET']) def blockchain(): ''' Shoots back the blockchain, which in our case, is a json list of hashes with the block information which is: index timestamp data hash prev_hash ''' node_blocks = sync.sync() #regrab the nodes if they've changed # Convert our blocks into dictionaries # so we can send them as json objects later python_blocks = [] for block in node_blocks: python_blocks.append(block.__dict__()) json_blocks = json.dumps(python_blocks) return json_blocks if __name__ == '__main__': node.run() 

Run this code, go to localhost: 3000 / blockchain.json and see the current block.

Step 4 - “Mining”, also known as block creation

Now there is only the genesis of the block, but if we have more data that needs to be stored and distributed, we need a way to include it in the new block. The question is how to create a new block and connect it with the previous one.

Satoshi describes it as follows in the bitcoin whitepaper. Note that the “timestamp server” is called a “node”.

“Let's start the description of our solution with the timestamp server. His job is to hash a block of data on which to put a timestamp, and open publication of this hash ... Timestamp shows that at this moment specific data existed and therefore fell into a block hash. Each hash includes the previous timestamp: this is how the chain is built, where the next link strengthens all previous ones. ”

Screenshot of the image attached under the description:



The main idea of ​​the section is to connect the blocks, if necessary, we create a hash of information about the new block, including the block creation time, the previous block hash, and the information in the block itself. I will call all this information a “block header”. Thus, we can check the correctness of the block, counting all the hashes in front of it, confirming the sequence.

In this case, the header that I create combines the string values ​​into one huge string. I have included the following data:

1. An index showing how the block is;
2. The hash of the previous block;
3. Data is just random strings. For bitcoin, they are called Merkle root and contain information about transactions;
4. Timestamp mining this unit.

 def generate_header(index, prev_hash, data, timestamp): return str(index) + prev_hash + data + str(timestamp) 

Let me explain one thing - the union of information lines is not mandatory for creating a header. The requirement is that everyone knows how to generate a block header and a hash of the previous block inside it. This is done so that everyone can verify the correctness of the hash in the new block and confirm the connection between the two blocks.

Bitcoin headers are much harder to concatenate strings. It uses data and time hashes and is tied to how data is located in memory. But in our case, string concatenation is sufficient.

Now we have a header and we can calculate the validity of the hash. I will use a method that is different from the Bitcoin method, but I still run the block header through the sha256 function.

 def calculate_hash(index, prev_hash, data, timestamp, nonce): header_string = generate_header(index, prev_hash, data, timestamp, nonce) sha = hashlib.sha256() sha.update(header_string) return sha.hexdigest() 

To mine a block, we use the function above to get a hash, put it in a new block and save this block in the chaindata directory.

 node_blocks = sync.sync() def mine(last_block): index = int(last_block.index) + 1 timestamp = date.datetime.now() data = "I block #%s" % (int(last_block.index) + 1) #random string for now, not transactions prev_hash = last_block.hash block_hash = calculate_hash(index, prev_hash, data, timestamp) block_data = {} block_data['index'] = int(last_block.index) + 1 block_data['timestamp'] = date.datetime.now() block_data['data'] = "I block #%s" % last_block.index block_data['prev_hash'] = last_block.hash block_data['hash'] = block_hash return Block(block_data) def save_block(block): chaindata_dir = 'chaindata' filename = '%s/%s.json' % (chaindata_dir, block.index) with open(filename, 'w') as block_file: print new_block.__dict__() json.dump(block.__dict__(), block_file) if __name__ == '__main__': last_block = node_blocks[-1] new_block = mine(last_block) save_block(new_block) 

Done! But with this type of block creation, anyone with the fastest CPU will be able to create the longest circuits that other nodes consider correct. We need a way to reduce the speed of block creation and confirmation before moving to the next block.

Step 5 - Proof of Work

To reduce the speed I use Proof of performance, like Bitcoin. Proof of ownership share is another method used in blockchains to achieve consensus, but in this case I will take advantage of the work.

The way to do this is to set the requirements for the block hash structure. As in the case of bitcoin, you need to make sure that the hash starts with a certain number of zeros, before proceeding to the next. And for this you need to add additional information to the header - a randomly selected number (nonce).

 def generate_header(index, prev_hash, data, timestamp, nonce): return str(index) + prev_hash + data + str(timestamp) + str(nonce) 

Now the mining function is configured to create a hash, but if the block hash does not contain enough zeros, we increase the nonce value, create a new header, calculate a new hash and check if there are enough zeros.

 NUM_ZEROS = 4 def mine(last_block): index = int(last_block.index) + 1 timestamp = date.datetime.now() data = "I block #%s" % (int(last_block.index) + 1) #random string for now, not transactions prev_hash = last_block.hash nonce = 0 block_hash = calculate_hash(index, prev_hash, data, timestamp, nonce) while str(block_hash[0:NUM_ZEROS]) != '0' * NUM_ZEROS: nonce += 1 block_hash = calculate_hash(index, prev_hash, data, timestamp, nonce) block_data = {} block_data['index'] = int(last_block.index) + 1 block_data['timestamp'] = date.datetime.now() block_data['data'] = "I block #%s" % last_block.index block_data['prev_hash'] = last_block.hash block_data['hash'] = block_hash block_data['nonce'] = nonce return Block(block_data) 

Fine. The new block contains a valid nonce value, so other nodes can confirm the hash. We can generate, save and distribute the rest of the block.

Conclusion

That's all! For now. There are still a lot of questions and features in the blockchains, which I did not explain.

For example, how to use other nodes? How do nodes transfer data when included in a block? Are there any other ways to store data besides huge rows of data?
The answers to these questions will be found in the following parts of this series of posts, as soon as I find the answers to them myself. You can write my wishes on the content to me on twitter, in the comments to the post or via the feedback form!

Thanks to my sister Sarah for asking questions about the blockchains and help in editing the post!

THE END

Comments, questions, as always, are welcome both here and at the open door.