Blockchain on Go. Part 3: Permanent Memory and Command Line Interface

Content

1. Blockchain on Go. Part 1: Prototype
2. Blockchain on Go. Part 2: Proof-of-Work
3. Blockchain on Go. Part 3: Permanent Memory and Command Line Interface
4. Blockchain on Go. Part 4: Transactions, Part 1
5. Blockchain on Go. Part 5: Addresses
6. Blockchain on Go. Part 6: Transactions, Part 2
7. Blockchain on Go. Part 7: Network

Introduction

In the previous part, we built a blockchain with a PoW system and the possibility of mining. Our implementation is getting closer to a fully functional blockchain, but it still lacks some important functions. Today we will start storing the blockchain in the database, then we will create a command line interface for operations with the blockchain. In essence, the blockchain is a distributed database. For now, we’ll omit the “distributed” and focus on the “database”.

Database selection

So far, we do not have a database in the implementation, we simply create blocks when the program is started and store them in memory. We cannot reuse or share our blockchain with others, so we need to save it to disk.

What database do we need? In fact, any will do. In Bitcoin Paper, nothing is said about a specific database, so the choice is up to the developer. Bitcoin Core , which was originally published by Satoshi Nakamoto and which is currently the reference implementation of Bitcoin, uses LevelDB (although it was only presented to the client in 2012). And we will use ...
')

BoltDB

Because:

1. It is simple and minimal.
2. It is implemented on Go
3. She does not need to start the server
4. It allows us to build the necessary data structures.

From BoltDB README :

Bolt is simply a key-value repository inspired by the Howard Chu LMDB project . The goal of the project is to provide a simple, fast and reliable database for projects that do not require a full-fledged database server, such as Postgres or MySQL.

Since Bolt is intended to be used as such a low-level element of functionality, simplicity is key. The API will be small and focus only on getting values ​​and setting values. It's all!

Sounds perfect for our needs! Take a moment to review the base.

BoltDB is a “key-value” repository, which means that there are no tables, like in relational DBMS (MySQL, PostgreSQL, etc.), there are no rows or columns. Instead, data is stored in key-value pairs (as in the Golang map). Pairs are stored in “baskets”, which are designed to group similar pairs (like tables in relational DBMS). Thus, to get the value, you need to know the basket and the key.

The important thing about BoltDB is that there are no data types here: keys and values ​​are byte arrays. Since we store Go structures (in particular, Block ), we must serialize them, that is, implement a mechanism for translating the structure into a byte array and restoring it back from the array. We will use encoding / gob for this, although JSON, XML, Protocol Buffers also suitable. We use encoding/gob because it is simple and it is part of the standard Go library.

Database structure

Before we start implementing persistent logic, we must decide how we will store our data in the database. And for this we will use the way that Bitcoin Core is used.

If simply, then Bitcoin Core uses two "baskets" for data storage.

1. blocks stores metadata describing all the blocks in a chain.
2. chainstate stores the chain state, which represents all unspent transaction outputs and some metadata

Also blocks are stored as separate files on disk. This is done to improve performance: reading one block does not require loading all (or some) into memory. This we will not implement.

In the blocks pair key->value is:

1. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->

2. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->

3. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->

4. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->

5. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->

6. 'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->


'b' + 32- -> 'f' + 4- -> 'l' -> 4- : 'R' -> 1- boolean : 'F' + 1- + -> 1 boolean: , 't' + 32- ->
In chainstate pairs, key->value is:

1. 'c' + 32- -> 'B' -> 32- : ,

2. 'c' + 32- -> 'B' -> 32- : ,


'c' + 32- -> 'B' -> 32- : ,
(A detailed explanation can be found here )

Since we have no transactions so far, we will only make cart blocks . In addition, as mentioned above, we will store the entire database in one file, without storing the blocks in separate files. Therefore, we do not need anything related to file numbers. Therefore, the key->value pairs that we will use are:

1. 32-byte block hash -> block structure (serialized)
2. 'l' -> hash of the last block in the chain

This is all we need to know to implement the mechanism of constancy (persistence).

Serialization

As stated earlier, in BoltDB, values ​​can only be of []byte type, and we want to store the Block structure in the database. We will use encoding/gob to serialize the structures.

Let's implement the Serialize method for Block (error handling is omitted for brevity)

 func (b *Block) Serialize() []byte { var result bytes.Buffer encoder := gob.NewEncoder(&result) err := encoder.Encode(b) return result.Bytes() } 

Everything is simple: at the beginning, we declare the buffer where the serialized data will be stored, then we initialize the gob encoder and encode the block, return the result as an array of bytes.

Now we need a deserialization function that takes an array of bytes as input and returns Block . This will not be a method, but an independent function:

 func DeserializeBlock(d []byte) *Block { var block Block decoder := gob.NewDecoder(bytes.NewReader(d)) err := decoder.Decode(&block) return &block } 

That's all we need for serialization.

Persistence

Let's start with the NewBlockchain function. Now she creates a new copy of Blockchain and adds a genesis block to it. We want to do the following:

1. Open DB file
2. Check if the blockchain is saved there.
3. If he is there:
   1. Create a new Blockchain instance
   2. Set the tip of the Blockchain instance to the hash of the last block stored in the database
   
   
4. If there is no existing blockchain
   
   
   1. Create genesis block
   2. Save to DB
   3. Save Genesis Hash as Hash of Last Last Block
   4. Create a new Blockchain instance with a tip indicating the genesis of the block

In code, it looks like this:

 func NewBlockchain() *Blockchain { var tip []byte db, err := bolt.Open(dbFile, 0600, nil) err = db.Update(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(blocksBucket)) if b == nil { genesis := NewGenesisBlock() b, err := tx.CreateBucket([]byte(blocksBucket)) err = b.Put(genesis.Hash, genesis.Serialize()) err = b.Put([]byte("l"), genesis.Hash) tip = genesis.Hash } else { tip = b.Get([]byte("l")) } return nil }) bc := Blockchain{tip, db} return &bc } 

Let's sort the code in parts.

 db, err := bolt.Open(dbFile, 0600, nil) 

This is the standard way to open a BoltDB file. Please note that it will not return an error if there is no file.

 err = db.Update(func(tx *bolt.Tx) error { ... }) 

In BoltDB, database operations are performed as part of a transaction. There are two types of transactions: read-only and read-write. Here we open the read-write transaction (db.Update(...)) , because we plan to put the genesis block in the database.

 b := tx.Bucket([]byte(blocksBucket)) if b == nil { genesis := NewGenesisBlock() b, err := tx.CreateBucket([]byte(blocksBucket)) err = b.Put(genesis.Hash, genesis.Serialize()) err = b.Put([]byte("l"), genesis.Hash) tip = genesis.Hash } else { tip = b.Get([]byte("l")) } 

This is the core of the function. Here we get a basket that stores our blocks: if it exists, then we read the key l from it, if it does not exist, then we generate a genesis block, create a basket, save the block in it and update the key l , which stores the hash of the last block in the chain.

Also notice the new way to create a Blockchain :

 bc := Blockchain{tip, db} 

We do not store all the blocks, instead we store only the tip of the chain. We also store the connection to the database, because we want to open it once and keep it open while the program is running. This is how the Blockchain structure looks like now:

 type Blockchain struct { tip []byte db *bolt.DB } 

The next thing we want to change is the AddBlock method: adding blocks to a chain is no longer as simple as adding an element to an array. From now on, we will store the blocks in the database:

 func (bc *Blockchain) AddBlock(data string) { var lastHash []byte err := bc.db.View(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(blocksBucket)) lastHash = b.Get([]byte("l")) return nil }) newBlock := NewBlock(data, lastHash) err = bc.db.Update(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(blocksBucket)) err := b.Put(newBlock.Hash, newBlock.Serialize()) err = b.Put([]byte("l"), newBlock.Hash) bc.tip = newBlock.Hash return nil }) } 

Consider the code bit by bit:

 err := bc.db.View(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(blocksBucket)) lastHash = b.Get([]byte("l")) return nil }) 

This is a different (read-only) type of transaction BoltDB. Here we get the hash of the last block from the database to use to mine the hash of a new block.

 newBlock := NewBlock(data, lastHash) b := tx.Bucket([]byte(blocksBucket)) err := b.Put(newBlock.Hash, newBlock.Serialize()) err = b.Put([]byte("l"), newBlock.Hash) bc.tip = newBlock.Hash 

After mining a new block, we save the serialized view to the database and update the key l , which now stores the hash of the new block.

Done! It was not difficult, was it?

Checking blockchain

All new blocks are now stored in the database, so we can rediscover the blockchain and add a new block to it. But after implementing this, we lose one useful feature: we cannot print blocks, because we no longer store them in an array. Let's fix it.

BoltDB allows you to go through all the keys in the basket, but all the keys are stored in the sort order bytes, and we want the blocks to be printed in the order in which they are placed in the blockchain. Also, since we do not want to load all the blocks into memory (our blockchain can be very huge), then we will read them one by one. For this purpose, we need a blockchain iterator:

 type BlockchainIterator struct { currentHash []byte db *bolt.DB } 

An iterator will be created every time we want to iterate over the blocks in the blockchain and it will store the hash of the current iteration block and the connection to the database. Because of the latter, the iterator is logically bound to the blockchain (this is a Blockchain instance that stores the connection to the database) and, thus, is created in the Blockchain method:

 func (bc *Blockchain) Iterator() *BlockchainIterator { bci := &BlockchainIterator{bc.tip, bc.db} return bci } 

Note that the iterator first points to the tip of the blockchain, so the blocks will be obtained from top to bottom, from the newest to the oldest. In fact, the choice of the tip means “voting” for the blockchain . A blockchain can have several branches and the longest of them is considered the main one. After receiving the tip (it can be any block in the blockchain), we can recreate the whole blockchain and find its length, and the work necessary to build it. This fact also means that the tip is a kind of blockchain identifier.

BlockchainIterator does only one thing: returns the next block from the blockchain.

 func (i *BlockchainIterator) Next() *Block { var block *Block err := i.db.View(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(blocksBucket)) encodedBlock := b.Get(i.currentHash) block = DeserializeBlock(encodedBlock) return nil }) i.currentHash = block.PrevBlockHash return block } 

That's all about the database!

Command Line Interface (CLI)

So far, our implementation does not provide us with any interface for interacting with the program: we simply executed NewBlockchain, bc.AddBlock in main . It's time to improve it! We want to have such commands:

 blockchain_go addblock "Pay 0.031337 for a coffee" blockchain_go printchain 

All command line related operations will be processed by the CLI structure.

 type CLI struct { bc *Blockchain } 

The “entry point” of the structure is the Run function Run

 func (cli *CLI) Run() { cli.validateArgs() addBlockCmd := flag.NewFlagSet("addblock", flag.ExitOnError) printChainCmd := flag.NewFlagSet("printchain", flag.ExitOnError) addBlockData := addBlockCmd.String("data", "", "Block data") switch os.Args[1] { case "addblock": err := addBlockCmd.Parse(os.Args[2:]) case "printchain": err := printChainCmd.Parse(os.Args[2:]) default: cli.printUsage() os.Exit(1) } if addBlockCmd.Parsed() { if *addBlockData == "" { addBlockCmd.Usage() os.Exit(1) } cli.addBlock(*addBlockData) } if printChainCmd.Parsed() { cli.printChain() } } 

We use the standard flag package for parsing command line arguments.

 addBlockCmd := flag.NewFlagSet("addblock", flag.ExitOnError) printChainCmd := flag.NewFlagSet("printchain", flag.ExitOnError) addBlockData := addBlockCmd.String("data", "", "Block data") 

To begin with, we create two addblock and printchain , then add the -data flag to the first one. printchain does not require any flags.

 switch os.Args[1] { case "addblock": err := addBlockCmd.Parse(os.Args[2:]) case "printchain": err := printChainCmd.Parse(os.Args[2:]) default: cli.printUsage() os.Exit(1) } 

Then we check the command specified by the user and parse the associated subcommand.

 if addBlockCmd.Parsed() { if *addBlockData == "" { addBlockCmd.Usage() os.Exit(1) } cli.addBlock(*addBlockData) } if printChainCmd.Parsed() { cli.printChain() } 

Next, we check which subcommand we use, and start the associated function.

 func (cli *CLI) addBlock(data string) { cli.bc.AddBlock(data) fmt.Println("Success!") } func (cli *CLI) printChain() { bci := cli.bc.Iterator() for { block := bci.Next() fmt.Printf("Prev. hash: %x\n", block.PrevBlockHash) fmt.Printf("Data: %s\n", block.Data) fmt.Printf("Hash: %x\n", block.Hash) pow := NewProofOfWork(block) fmt.Printf("PoW: %s\n", strconv.FormatBool(pow.Validate())) fmt.Println() if len(block.PrevBlockHash) == 0 { break } } } 

This code is similar to the one that was before. The only difference is that now we use the BlockchainIterator to iterate over the blocks in the blockchain.

Also, don't forget to change the main function accordingly:

 func main() { bc := NewBlockchain() defer bc.db.Close() cli := CLI{bc} cli.Run() } 

Note that a new Blockchain is created regardless of which command line arguments were passed.

That's all! Let's check that everything works as we expect:

 $ blockchain_go printchain No existing blockchain found. Creating a new one... Mining the block containing "Genesis Block" 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b Prev. hash: Data: Genesis Block Hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b PoW: true $ blockchain_go addblock -data "Send 1 BTC to Ivan" Mining the block containing "Send 1 BTC to Ivan" 000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13 Success! $ blockchain_go addblock -data "Pay 0.31337 BTC for a coffee" Mining the block containing "Pay 0.31337 BTC for a coffee" 000000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148 Success! $ blockchain_go printchain Prev. hash: 000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13 Data: Pay 0.31337 BTC for a coffee Hash: 000000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148 PoW: true Prev. hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b Data: Send 1 BTC to Ivan Hash: 000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13 PoW: true Prev. hash: Data: Genesis Block Hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b PoW: true 

(the sound of opening a beer can )

Links

Original article
The first part of the series of articles
Sources
Bitcoin Core Data Storage
BoltDB
encoding / gob
flag