Exonum 0.3 - what we have improved in the new version of the framework for the development of blockchains
We continue to develop Exonum . This time we focused on two things: we completely transferred data storage to RocksDB, at the same time stopped supporting LevelDB, and rewrote the network code to Tokio. Why: these solutions allowed us to increase the efficiency of data storage and increase the performance of the code in the network.
We talked about the transition from LevelDB to RocksDB in the article about the release of the previous version of the framework. Therefore, in today's announcement, we would like to dwell in greater detail on the changes that came with Tokio, to tell how we implemented asynchronous event handling, and to note other improvements.
/ Exonum
')
Tokio is an event loop handler that is specially designed for asynchronous programming. It is based on asynchronous I / O (I / O), which ensures continuous work with events in the stream. Individual events are serviced in the background until the end result is obtained.
The parameter that provides asynchronous event management is called future . This value, which will be calculated in the future, but is currently not known. Futures are composite objects and can include chains of events that implement the business logic of processes.
For example, in the context of Exonum, it can be represented as follows:
Connect to node "B" -> Process the bytes of node "B" and break them into messages -> Forward each message to the channel of the node
In addition to futures, Tokio also has a base component for asynchronous I / O - stream. This is a stream of bytes that is converted to other elements. While the future returns only one final result, stream can return several results until it is fully executed. In other words, stream operates with a series of events, which makes it effective when working with complex interaction schemes.
For example, in Exonum TCP stream will be converted to Exonum messages. This stream is sent to the consensus code via the appropriate channel, and after merging them, the code selects the messages it needs from the network and processes them.
Another significant improvement was the ability of the nodes to work with three different queues instead of the one that existed in the network code on Mio. Globally, the network manages three main types of events when the consensus algorithm is executed: incoming transactions, messages, and a timer.
Previously, transactions and messages overloaded one queue, and the timer events remained outside of it. Accordingly, the new rounds of the consensus algorithm did not start, and he simply stalled. The presence of additional queues dramatically increased the stability of nodes.
Similarly, for the new code, work was organized in two streams: one for servicing external network events (responses to network messages), and the second for the consensus itself. However, there is also a separate stream for transferring transactions received via the REST API. As a result, this increased the performance and stability of the network: new blocks are received at a uniform interval of 0.5 seconds, while, if necessary, the rate of adoption of the block can be reduced.
The pull-up speed has doubled, that is, the time it takes for a lagging node to get its missing blocks has decreased. For example, in a network with 4 nodes, the block acceptance rate with parallel pulling was previously 200–300 ms. Translation of only the lagging node to the new code led to the fact that the interval was reduced to 100 ms. It is expected that if you adapt the entire network, the indicators will become even better. The statistics above are for empty blocks and may differ for the network "under load".
We also carried out an approximate calculation of the effect of the described changes on the code performance in a network with 4 validators and one data center. The average processing speed when accepting blocks with a thousand transactions was:
Thus, with the transition to Tokio, the code in Exonum became more efficient and structured, which made it easier to maintain.
Another improvement was the introduction of a new index into the repository - SparseListIndex . It is an ordered structure (list of elements), which may contain omissions. This means that if you delete an arbitrary row in the list, the index will remain operable. For comparison, in ListIndex all entries are strictly numbered from 0 to n-1, where n is the number of elements in the list and can only be deleted from the end of the list.
A basic infrastructure has also been added to collect metrics and statistics in Exonum and services. The option is useful for evaluating the effectiveness of the site and will be finalized in future releases.
Finally, we would like to draw the attention of service developers to the following positions:
All the above listed changes need to be made to the existing service code in order to keep it up to date with respect to the framework.
/ Exonum
More about innovations - by reference . With questions to our team and suggestions for the development of the project, you can contact Gitter or GitHub .
We talked about the transition from LevelDB to RocksDB in the article about the release of the previous version of the framework. Therefore, in today's announcement, we would like to dwell in greater detail on the changes that came with Tokio, to tell how we implemented asynchronous event handling, and to note other improvements.
/ Exonum
')
Tokio is an event loop handler that is specially designed for asynchronous programming. It is based on asynchronous I / O (I / O), which ensures continuous work with events in the stream. Individual events are serviced in the background until the end result is obtained.
The parameter that provides asynchronous event management is called future . This value, which will be calculated in the future, but is currently not known. Futures are composite objects and can include chains of events that implement the business logic of processes.
For example, in the context of Exonum, it can be represented as follows:
Connect to node "B" -> Process the bytes of node "B" and break them into messages -> Forward each message to the channel of the node
In addition to futures, Tokio also has a base component for asynchronous I / O - stream. This is a stream of bytes that is converted to other elements. While the future returns only one final result, stream can return several results until it is fully executed. In other words, stream operates with a series of events, which makes it effective when working with complex interaction schemes.
For example, in Exonum TCP stream will be converted to Exonum messages. This stream is sent to the consensus code via the appropriate channel, and after merging them, the code selects the messages it needs from the network and processes them.
Another significant improvement was the ability of the nodes to work with three different queues instead of the one that existed in the network code on Mio. Globally, the network manages three main types of events when the consensus algorithm is executed: incoming transactions, messages, and a timer.
Previously, transactions and messages overloaded one queue, and the timer events remained outside of it. Accordingly, the new rounds of the consensus algorithm did not start, and he simply stalled. The presence of additional queues dramatically increased the stability of nodes.
Similarly, for the new code, work was organized in two streams: one for servicing external network events (responses to network messages), and the second for the consensus itself. However, there is also a separate stream for transferring transactions received via the REST API. As a result, this increased the performance and stability of the network: new blocks are received at a uniform interval of 0.5 seconds, while, if necessary, the rate of adoption of the block can be reduced.
The pull-up speed has doubled, that is, the time it takes for a lagging node to get its missing blocks has decreased. For example, in a network with 4 nodes, the block acceptance rate with parallel pulling was previously 200–300 ms. Translation of only the lagging node to the new code led to the fact that the interval was reduced to 100 ms. It is expected that if you adapt the entire network, the indicators will become even better. The statistics above are for empty blocks and may differ for the network "under load".
We also carried out an approximate calculation of the effect of the described changes on the code performance in a network with 4 validators and one data center. The average processing speed when accepting blocks with a thousand transactions was:
- 7318 transactions per second for Mio and LevelDB code;
- 20237 transactions per second for Tokio and LevelDB code;
- 31571 transactions per second for Tokio and RocksDB code.
Thus, with the transition to Tokio, the code in Exonum became more efficient and structured, which made it easier to maintain.
Another improvement was the introduction of a new index into the repository - SparseListIndex . It is an ordered structure (list of elements), which may contain omissions. This means that if you delete an arbitrary row in the list, the index will remain operable. For comparison, in ListIndex all entries are strictly numbered from 0 to n-1, where n is the number of elements in the list and can only be deleted from the end of the list.
A basic infrastructure has also been added to collect metrics and statistics in Exonum and services. The option is useful for evaluating the effectiveness of the site and will be finalized in future releases.
Finally, we would like to draw the attention of service developers to the following positions:
- The events_pool_capacity field in MemoryPoolConfig has been replaced with a new EventsPoolCapacity configuration;
- NodeBuilder now uses the ServiceFactory as an object, not as a type;
- The signature of the gen_prefix function in the schema module has been changed;
- The index constructor has changed. In connection with the use of column families available in RocksDB, it has become possible to define indexes using string names, as is usually implemented in other databases (for example, Transactions, Wallets, etc.).
All the above listed changes need to be made to the existing service code in order to keep it up to date with respect to the framework.
/ Exonum
More about innovations - by reference . With questions to our team and suggestions for the development of the project, you can contact Gitter or GitHub .
Source: https://habr.com/ru/post/342970/
All Articles