zetcd from CoreOS: Replacing ZooKeeper with ... storage etcd
Last week, CoreOS was pleased with another open source project - zetcd . In fact, it was known about him since last year, but now the first release took place, which translated the product into beta testing status - declared the product is ready for serious tests before being released into the world of production. The authors have positioned zetcd as a ready-made replacement for ZooKeeper within such distributed / cluster solutions as Mesos, Apache Kafka and Apache Drill. Even the fact that etcd offers a “flat” key-value storage against the competitor’s hierarchical approach does not hinder their attitude. How did they come to this?
Apache ZooKeeper is unlikely to need a special presentation, but nonetheless: it is a key-value keystore (KV) written in Java. It is characterized by a hierarchical namespace and is focused on application in distributed applications, where it acts as a centralized service for storing configurations and other service information.
According to the authors of ZooKeeper, this project appeared (initially, by the way, inside Hadoop) as a response to repeated attempts of developers of distributed systems to create their own repositories, which invariably led to new challenges and difficulties in support. And I must admit that in the ten years of its existence, ZooKeeper has managed to gain a foothold in the market, becoming an important component for such open source projects as Apache Drill, Apache Hadoop YARN, Apache HBase, Apache Kafka, Apache Solr, Mesos, Norbert. (By the way, there are examples of those who eventually abandoned ZooKeeper: Juju and Neo4j .) Of course, some developers of closed systems and applications that required distributed KV storage also chose ZooKeeper. Finally, this solution has become popular in the world of microservices, where it is also used as a service discovery service , although there is no problem without it .
Of course, Apache ZooKeeper is not the only solution in its niche. There are, for example, Consul , offering KV-storage and placing particular emphasis on the mentioned Service Discovery, as well as Doozer , focused on high availability (to the detriment of scalability and performance), and the main “character” of this article - etcd . (For those interested in the difference of these solutions, the articles “ Consul vs. ZooKeeper, doozerd, etcd ” according to HashiCorp, “ Exploring Performance of etcd, Zookeeper and Consul Consistent Key-value Datastores ” according to CoreOS may be useful.)
')
The etcd storage was created in CoreOS for critical data in distributed systems, written in the Go language and seeks to provide: a) simplicity (simple API, accessible through gRPC), b) security (automatic TLS), c) performance (the authors promise 10,000 operations records per second), d) reliability (thanks to the Raft consensus algorithm). All this, seasoned with the interests and enthusiasm of CoreOS, allowed the new solution to gain some weight: it suffices to mention that etcd became a permanent repository of REST API objects in Kubernetes.
The further move of CoreOS is not surprising: if many people in the world use ZooKeeper, and the company has an interest in promoting its competing product, then why not make the migration as simple as possible ... And since the data model and client protocol etcd differ from that used in ZooKeeper and will not change for the sake of this compatibility, the solution is obvious - release a layer that will redirect all application requests that are addressed to ZooKeeper, in the cluster etcd. Fortunately, “a rather expressive API in etcd v3 allowed to emulate the ZooKeeper data model on the client side using a normal proxy”. Convenience for developers is that there is no need to modify applications at all: it is enough to replace the ZooKeeper installation with etcd and add a new proxy server in front of it - the same zetcd.
Understand its device will help the project announcement and small practical experiments. So, zetcd (written in the Go language, as well as etcd) was created in order to receive application requests sent to ZooKeeper, and convert them into operations corresponding to the etcd data model and executed in this repository.
To run zetcd, only the Go compiler is required. Installing etcd and zetcd:
Note : According to the experience of installations on different versions of Ubuntu, it is best (to install the latest versions of all applications from the CoreOS Git repositories) to have Go 1.8 installed, which for 16.04 can be taken from ppa: longsleep / golang-backports.
Run etcd (by default it will bind to localhost: 2379) and zetcd:
Installing zkctl (CLI utility for simple operations in ZooKeeper) and test commands:
Obviously, a new node was created in the root of the ZooKeeper tree (
To answer this question, you need to better understand how zetcd transforms the ZooKeeper hierarchical data model into a format that is understandable to etcd. The ZooKeeper tree structure fits into a flat KV-space in such a way that for each record KV-pairs are created with metadata, the keys of which are the full path that also contains the name of the parameter and the nesting level:
That is, for each key from ZooKeeper, metadata about its revision, version and access rights is stored. They are simplified compared to the original ZNode due to the specifics of etcd (lack of directories in the tree per se, role-based authentication instead of an ACL, etc.), but fully describe the node. For example:
You can check the physical organization of the data in action using the etcdctl console client:
Note : if you want to repeat the last operation, to see with your own eyes how ZNode data is stored in etcd, then you need to collect zetcd with
Considering the performance is not the first necessity, but an important component of success for zetcd, the developers took care of creating a simple benchmark utility - zkboom . CoreOS compared ZooKeeper 3.4.10 and dev-branches etcd with writing and reading 128-byte KV-pairs, limiting the number of requests to 2500 per second and increasing the number of simultaneous clients. All this was done on “two modern Linux machines connected by a gigabit switch; one was running proxy and server software, and the other generated client requests. ”
As you can see, in the creation of keys, the delay in zetcd reaches ~ 2-3 times worse values ​​than in ZooKeeper, and in reading - ~ 1.5 times. This is not to say that these are grandiose results (rather, they are quite understandable, given the need to work with a lot of additional keys to emulate ZNode metadata), but you can’t call them very bad, given the specifics of the proxy server's purpose.
By the way, other utilities created for testing the performance of ZooKeeper itself should already work with a bunch of etcd and zetcd, which opens up a large field for independent experiments and conclusions.
The first public release of zetcd - v0.0.1 - took place last week. According to the authors, even though there is still room for improvements in the performance of zetcd, soon the product can become a ready replacement for ZooKeeper in various applications, which in addition to well-known projects include in-house development categories, the authors of which want to - or for reasons to leave ZooKeeper and make it a "little blood". Finally, an interesting feature of this replacement appears in the context of the use of etcd Operator for Kubernetes , which adds automated updates, backups and other features to this ZooKeeper compatible repository.
Background: Apache ZooKeeper
Apache ZooKeeper is unlikely to need a special presentation, but nonetheless: it is a key-value keystore (KV) written in Java. It is characterized by a hierarchical namespace and is focused on application in distributed applications, where it acts as a centralized service for storing configurations and other service information.
According to the authors of ZooKeeper, this project appeared (initially, by the way, inside Hadoop) as a response to repeated attempts of developers of distributed systems to create their own repositories, which invariably led to new challenges and difficulties in support. And I must admit that in the ten years of its existence, ZooKeeper has managed to gain a foothold in the market, becoming an important component for such open source projects as Apache Drill, Apache Hadoop YARN, Apache HBase, Apache Kafka, Apache Solr, Mesos, Norbert. (By the way, there are examples of those who eventually abandoned ZooKeeper: Juju and Neo4j .) Of course, some developers of closed systems and applications that required distributed KV storage also chose ZooKeeper. Finally, this solution has become popular in the world of microservices, where it is also used as a service discovery service , although there is no problem without it .
And here etcd?
Of course, Apache ZooKeeper is not the only solution in its niche. There are, for example, Consul , offering KV-storage and placing particular emphasis on the mentioned Service Discovery, as well as Doozer , focused on high availability (to the detriment of scalability and performance), and the main “character” of this article - etcd . (For those interested in the difference of these solutions, the articles “ Consul vs. ZooKeeper, doozerd, etcd ” according to HashiCorp, “ Exploring Performance of etcd, Zookeeper and Consul Consistent Key-value Datastores ” according to CoreOS may be useful.)
')
The etcd storage was created in CoreOS for critical data in distributed systems, written in the Go language and seeks to provide: a) simplicity (simple API, accessible through gRPC), b) security (automatic TLS), c) performance (the authors promise 10,000 operations records per second), d) reliability (thanks to the Raft consensus algorithm). All this, seasoned with the interests and enthusiasm of CoreOS, allowed the new solution to gain some weight: it suffices to mention that etcd became a permanent repository of REST API objects in Kubernetes.
The further move of CoreOS is not surprising: if many people in the world use ZooKeeper, and the company has an interest in promoting its competing product, then why not make the migration as simple as possible ... And since the data model and client protocol etcd differ from that used in ZooKeeper and will not change for the sake of this compatibility, the solution is obvious - release a layer that will redirect all application requests that are addressed to ZooKeeper, in the cluster etcd. Fortunately, “a rather expressive API in etcd v3 allowed to emulate the ZooKeeper data model on the client side using a normal proxy”. Convenience for developers is that there is no need to modify applications at all: it is enough to replace the ZooKeeper installation with etcd and add a new proxy server in front of it - the same zetcd.
Basics of zetcd
Understand its device will help the project announcement and small practical experiments. So, zetcd (written in the Go language, as well as etcd) was created in order to receive application requests sent to ZooKeeper, and convert them into operations corresponding to the etcd data model and executed in this repository.
To run zetcd, only the Go compiler is required. Installing etcd and zetcd:
$ go get github.com/coreos/etcd/cmd/etcd $ go get github.com/coreos/zetcd/cmd/zetcd Note : According to the experience of installations on different versions of Ubuntu, it is best (to install the latest versions of all applications from the CoreOS Git repositories) to have Go 1.8 installed, which for 16.04 can be taken from ppa: longsleep / golang-backports.
Run etcd (by default it will bind to localhost: 2379) and zetcd:
$ etcd & $ zetcd -zkaddr localhost:2181 -endpoints localhost:2379 & Installing zkctl (CLI utility for simple operations in ZooKeeper) and test commands:
$ go install github.com/coreos/zetcd/cmd/zkctl $ zkctl watch / & $ zkctl create /abc "test" $ zkctl get /abc 2017/05/23 21:57:44 Connected to 127.0.0.1:2181 2017/05/23 21:57:44 Authenticated: id=7587822333160445481, timeout=1000 [116 101 115 116] Stat: &{Czxid:3 Mzxid:14 Ctime:1495550621626193 Mtime:1495551461984432 Version:1 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:4 NumChildren:1 Pzxid:7} Obviously, a new node was created in the root of the ZooKeeper tree (
abc ), with which the test string was associated (116,111,115,116 — numbers of unicode characters confirming what happened). But this is the terminology of ZooKeeper, and what happened in etcd?Internal zetcd device
To answer this question, you need to better understand how zetcd transforms the ZooKeeper hierarchical data model into a format that is understandable to etcd. The ZooKeeper tree structure fits into a flat KV-space in such a way that for each record KV-pairs are created with metadata, the keys of which are the full path that also contains the name of the parameter and the nesting level:
/zk/[param]/[depth]/[full path] . Thus, viewing the keys by directories in ZooKeeper ( getChildren ) corresponds to viewing the list of keys by interval in etcd ( Range ), and inside the etcd itself there is additional information that forms the complete ZNode view:That is, for each key from ZooKeeper, metadata about its revision, version and access rights is stored. They are simplified compared to the original ZNode due to the specifics of etcd (lack of directories in the tree per se, role-based authentication instead of an ACL, etc.), but fully describe the node. For example:
/zk/key/001/abc- stores the value of the/abckey (the first level of nesting in the ZooKeeper tree);/zk/mtime/002/abc/xyz- modification time of the/abc/xyzkey (second level of nesting).
You can check the physical organization of the data in action using the etcdctl console client:
$ go get github.com/coreos/etcd/etcdctl $ export ETCDCTL_API=3 $ etcdctl get --prefix /zk /zk/acl/001/abc - ACL PermsScheme ID >worldanyone /zk/count/001/abc /zk/ctime/001/abc Ţ„ /zk/cver/001/ /zk/cver/001/abc /zk/err-node 1 /zk/key/001/abc test /zk/mtime/001/abc Ţ„ /zk/ver/001/abc Note : if you want to repeat the last operation, to see with your own eyes how ZNode data is stored in etcd, then you need to collect zetcd with
-tags path_debug (i.e. go get -u -v -tags path_debug github.com/coreos/zetcd/cmd/zetcd ), and you also need a version from the Git repository, which incorporates a fresh Pull Request # 54 , created as a result of my curiosity in Issue # 52 ).Performance zetcd
Considering the performance is not the first necessity, but an important component of success for zetcd, the developers took care of creating a simple benchmark utility - zkboom . CoreOS compared ZooKeeper 3.4.10 and dev-branches etcd with writing and reading 128-byte KV-pairs, limiting the number of requests to 2500 per second and increasing the number of simultaneous clients. All this was done on “two modern Linux machines connected by a gigabit switch; one was running proxy and server software, and the other generated client requests. ”
As you can see, in the creation of keys, the delay in zetcd reaches ~ 2-3 times worse values ​​than in ZooKeeper, and in reading - ~ 1.5 times. This is not to say that these are grandiose results (rather, they are quite understandable, given the need to work with a lot of additional keys to emulate ZNode metadata), but you can’t call them very bad, given the specifics of the proxy server's purpose.
By the way, other utilities created for testing the performance of ZooKeeper itself should already work with a bunch of etcd and zetcd, which opens up a large field for independent experiments and conclusions.
Conclusion
The first public release of zetcd - v0.0.1 - took place last week. According to the authors, even though there is still room for improvements in the performance of zetcd, soon the product can become a ready replacement for ZooKeeper in various applications, which in addition to well-known projects include in-house development categories, the authors of which want to - or for reasons to leave ZooKeeper and make it a "little blood". Finally, an interesting feature of this replacement appears in the context of the use of etcd Operator for Kubernetes , which adds automated updates, backups and other features to this ZooKeeper compatible repository.
Source: https://habr.com/ru/post/329224/
All Articles