Operations as objects

Not so long ago, I had to access the ZooKeeper repository from C ++ code. There was no decent C ++ wrapper for the libzookeeper sishna library, so I had to write it myself. In the process of implementation, I have significantly modified the approach of the authors of the java-library to build an API, and now I want to share with you the reasons and the results of the decisions made. Despite the keywords C ++ and ZooKeeper, the approach described in the article is suitable for organizing access to any repositories, and is fully implementable in languages ​​other than C ++.

Introduction

ZooKeeper is a fault-tolerant distributed database that presents data as a hierarchical set of nodes. Nodes can be created, modified, deleted, checked for their existence, and managed for their access rights. Some operations take additional options, for example, you can specify the version of the node to which the command applies. The multithreaded client ZooKeeper, which is discussed in this article, creates two additional streams - in one it performs all I / O operations, in the other it performs custom callbacks and monitors. Monitors are functions that are called by the client when a node changes state. For example, you can find out if a node exists and pass a function that will be called when the node is gone or appears. The rest of the details necessary for understanding the article, I will give as needed.
We needed ZooKeeper to coordinate the execution of tasks by a multitude of machines in several data centers.
')
Having started working on a C ++ library, I decided to make the API as close as possible to the Java client API , which has one large ZooKeeper class, providing one method for each operation on the nodes. However, the shortcomings of this approach were quickly discovered.

I wanted to have several options for each command:

• Standard asynchronous: we pass the request parameters and the callback function (callback) to the client. When the operation is completed, the client will call the provided function in a separate thread created by the ZooKeeper client.
• Asynchronous, returning object std::future . We pass the request parameters to the client, the client returns an object representing the asynchronous calculation. When we call the std::future::get method, we will be given control after the operation completes. If the operation fails, the call to std::future::get should throw an exception.
• Synchronous. The call is blocked until the operation is completed. Errors are translated into exceptions.

If the operation has N possible options (with monitor / without monitor, with version / without version, etc.) and M variants of execution, we are waiting for writing and support for N * M methods. For example, in the java-client there are 4 exists methods:

 Stat exists(String path, boolean watch) void exists(String path, boolean watch, AsyncCallback.StatCallback cb, Object ctx) Stat exists(String path, Watcher watcher) void exists(String path, Watcher watcher, AsyncCallback.StatCallback cb, Object ctx) 

If you want to have an option that returns future , you have to add 2 more methods. A total of 6 methods, and this is only for one operation! I considered this unacceptable.

Types rush to the rescue

After realizing the futility of the obvious way, I got the idea of ​​restructuring the API - I need to separate as much as possible the way the command is executed from the command itself. Each command needs to be arranged in the form of a separate type - a container of parameters.

In the client in this case, you need to implement only one method for asynchronous command execution:

 void run(Command cmd, Callback callback); 

It is enough to implement only the basic asynchronous version of the implementation; all other options can be implemented outside the client, using the asynchronous interface as the basis without harming the encapsulation at all .

So, for each operation we will get a separate class:

• CreateCmd
• DeleteCmd
• ExistsCmd
• ...

Each class will store all the parameters that are needed to execute the command. For example, the delete operation takes a required path and can optionally take a version of the data to which it applies. The exists operation also requires a path and can optionally take a function called when a node is deleted / created.

Here you can select some templates - for example, all commands must contain the path to the node, some can be applied to a specific version ( delete , setACL , setData ), some accept an additional callback monitor, or they can publish events to the session callback monitor. You can implement these "templates" in the form of impurities (mixins) , from which we, as from bricks, will assemble our teams. In total, I managed to see 3 impurities:

• Pathable - takes a path in the constructor and provides a method for getting the path.
• Versionable - stores the version and provides methods for specifying the version and getting the specified version.
• Watchable - stores and allows you to define a callback that is called when a node changes state.

For example, here’s the impurity code Versionable :

 template <typename SelfType> struct Versionable { explicit Versionable(Version version = AnyVersion) : version_(version) {} SelfType & setVersion(Version version) { this->version_ = version; return static_cast<SelfType &>(*this); } Version version() const { return this->version_; } private: Version version_; }; 

To setVersion return the setVersion base class Versionable , the curiously recurring template pattern technique is used here. Adding impurities to teams is as follows:

 struct DeleteCmd : Pathable, Versionable<DeleteCmd> { explicit DeleteCmd(std::string path); // other methods }; 

The next step is to determine which type of callback should correspond to each of the commands, because when different commands are completed, the values ​​of different types are transferred to the callbacks. In total, there are 7 types of callbacks:

 using VoidCallback = std::function<void(std::error_code const&)>; using StatCallback = std::function<void(std::error_code const&, Stat const&)>; using ExistsCallback = std::function<void(std::error_code const&, bool, Stat const&)>; using StringCallback = std::function<void(std::error_code const&, std::string)>; using ChildrenCallback = std::function<void(std::error_code const&, Children, Stat const&)>; using DataCallback = std::function<void(std::error_code const&, std::string, Stat const&)>; using AclCallback = std::function<void(std::error_code const&, AclVector, Stat const&)>; 

The easiest way to bind commands to callbacks is to require each team to determine the appropriate nested type CallbackType . This is not quite beautiful, as the team begins to guess that it will be performed asynchronously with a callback, and this is exactly what we tried to avoid. However, I chose this particular implementation because of its simplicity and the fact that the asynchronous implementation is basic, and the remaining options will be a superstructure above it.

Next, you need to write code that will execute our commands asynchronously. The easiest option is to place the responsibility for parameter packing and non-blocking command launch on the command classes themselves. This is also a bit contrary to the accepted philosophy, but it allows you to keep all the logic of asynchronous processing of commands in one place. If the next version of ZooKeeper contains a new command, it will be enough to add just one class to our library, the changes will be very local and backward compatible.

For the unity of the command interface, I decided to enter the abstract type Handle - a low-level descriptor that hides all implementation details from the library client (for example, the fact that the command libzookeeper used to execute commands). In C / C ++, this can be achieved by declaring a type, but not defining it in the library’s public header files:

 class Handle; 

How exactly the Handle class is implemented is not so important. For simplicity, we can assume that this is in fact zhandle_t from the libzookeeper library, and the implementation of our commands in secret from the user converts the pointer to our incomplete type into a pointer to zhandle_t .

Thus, in each class representing the command, an overloaded call operator appears.

 struct SomeCmd { using CallbackType = SomeCallbackType; void operator()(Handle *, CallbackType) const; }; 

I will not give the packing code of command parameters, since it is rather cumbersome and contains a lot of intermediate code and details not related to the essence of the article.

The method of running commands in the client class becomes quite simple:

 class Session { public: // other methods template <typename CmdType> void run(CmdType const& cmd, typename CmdType::CallbackType callback) { cmd(this->getHandle(), std::move(callback)); } private: Handle * getHandle(); }; 

Here we essentially delegate the launch of the team to the team itself, passing it a low-level descriptor.

It is important that the overloaded call operator is constant - the commands should not change their state when called. First, it will allow the use of temporary command objects in the following code:

 session.run(DeleteCmd("path").setVersion(knownVersion), myCallback); 

The same effect could be achieved by transferring commands by value and relying on move-semantics , but this would have made it necessary in some cases to create redundant copies.

Secondly, in this way we inform the person reading the code (and, partly, the compiler) that repeated execution of the same command should not lead to side effects that are not related to changes in the structure of the repository.

Now we can asynchronously perform all operations with all possible options, and for this we need only one method in the client - run .

Add asynchronous command execution with std::future

So, now it is the turn to realize that, for the sake of which everything was started - alternative variants of implementation.

To be able to asynchronously execute commands with the std::future object, I want to have a function that has the following signature:

 template<class CmdType> std::future<ResultOf<CmdType>> runAsync(Session &, CmdType const&); 

This function accepts a session and a command as input, returning a std::future object representing the result of the asynchronous command execution.

First you need to understand how to fit the command callback parameters into one value. This is what the ResultOf metafunction is ResultOf . There are several ways to indicate the correspondence of function parameters to returned values, I chose the simplest one - just write out every possible case as a separate specialization of the template class DeduceResult .

 template <typename CallbackType> struct DeduceResult; template <> struct DeduceResult<std::function<void(std::error_code const&)>> { using type = void; }; template <typename T> struct DeduceResult<std::function<void(std::error_code const&, T)>> { using type = typename std::decay<T>::type; }; template <typename T> struct DeduceResult<std::function<void(std::error_code const&, T, Stat const&)>> { using type = std::pair<typename std::decay<T>::type, Stat>; }; template <typename CmdType> using ResultOf = typename DeduceResult<typename CmdType::CallbackType>::type; 

The logic of DeduceResult is simple:

• If only the error code is transferred to the callback, then the result of the operation is void .
• If an additional parameter is passed to the callback, in addition to the error code, the result of the operation will be this parameter.
• If the link to the Stat object is also transferred to the callback, the result and the Stat object will be packed into std::pair .

ResultOf is a template synonym ( alias template , one of the nice features of C ++ 11) that passes to the DeduceResult type of callback defined in the command.

The use of the std::decay metafunction is noteworthy - some parameters are passed to the callback by reference, but we want to return them to customers by value, because objects can live on the stack and, if they are passed to another thread, references to them will already be destroyed by the time the client reads them.

Now you can do the implementation of the function runAsync . The implementation is almost obvious: you need to create the std::promise object of the required type, get the std::future object from it (by calling the std::promise::get_future() method), create a special callback that will receive the std::promise object and put in it the result or error of the callback. Then you just need to execute the command through the standard session interface with our callback. Since a promise should have a callback, it is logical to make a callback a function object containing a promise as a field. The resulting runAsync function runAsync looks like this:

 template <typename CmdType> std::future<ResultOf<CmdType>> runAsync(Session & session, CmdType const& cmd) { FutureCallback<ResultOf<CmdType>> cb; auto f = cb.getPromise().get_future(); session.run(cmd, std::move(cb)); return f; } 

The implementation of the FutureCallback function FutureCallback in many ways mirrors the logic we nested in the ResultOf metafunction. Based on the expected type of operation, we generate functions that pack our input arguments into an object and pass this object to a common (with a future object) state via promise::set_value or promise::set_exception .

 template <typename T> void setError(std::promise>T> & p, std::error_code const& ec) { //    codeToExceptionPtr  . //     ,    //     exception_ptr,   //  . p.set_exception(codeToExceptionPtr(ec)); } template <typename T> struct CallbackBase { //       promise   ,  //   std::function -    . // . n337 (20.8.11.2.1) using PromisePtr = std::shared_ptr<std::promise<T>>; PromisePtr promisePtr; CallbackBase() : promisePtr(std::make_shared<std::promise<T>>()) {} std::promise<T> & getPromise() { return *promisePtr.get(); } }; template <typename T> struct FutureCallback : CallbackBase<T> { void operator()(std::error_code const& ec, T value) { if (ec) { setError(this->getPromise(), ec); } else { this->getPromise().set_value(std::move(value)); } } }; template <> struct FutureCallback<void> : CallbackBase<void> { void operator()(std::error_code const& ec) { if (ec) { setError(this->getPromise(), ec); } else { this->getPromise().set_value(); } } }; template >typename T> struct FutureCallback<std::pair<T, Stat>> : CallbackBase<std::pair<T, Stat>> { void operator()(std::error_code const& ec, T data, Stat const& stat) { if (ec) { setError(this->getPromise(), ec); } else { this->getPromise().set_value( std::make_pair(std::move(data), stat)); } } }; 

Now we can use our function as follows:

 std::vector<std::future<std::string>> nodeNameFutures; //     for (const auto & name : {"/node1", "/node2", "/node3"}) { nodeNameFutures.emplace_back(runAsync(session, CreateCmd(name))); } //     for (auto & f : nodeNameFutures) { f.wait(); } 

Writing such a code using callback functions is not a very pleasant thing. Using the std::future mechanism greatly simplifies such tasks. For example, I used this mechanism to implement the function of recursive deletion of a subtree.

Other embodiments

Synchronous command execution we get almost for free:

 template <typename CmdType> ResultOf<CmdType> runSync(Session & session, CmdType const& cmd) { return runAsync(session, cmd).get(); } 

You can come up with many more different embodiments. For example, you can quite easily write a function that will rerun a command when a connection error occurs, or it records the parameters, the beginning and end of command execution (for example, using the chromium trace_event API ) to debug and analyze performance.

In fact, we have a primitive and strongly limited version of the aspect-oriented paradigm . We can perform additional actions before the launch and after the completion of commands, localizing the logic within one function - the "aspect".

Conclusion

Transforming operations from storage into methods reduced the coherence of the code and significantly reduced its number, saving a lot of effort in implementation and debugging.

The described approach, of course, is not something fundamentally new. At a minimum, a similar practice is used in the HBase Java client .

There are drawbacks to this method - it generates quite a few classes, and it becomes a bit more difficult for clients to explore the library interface — operations are not grouped together in a class interface, but are separated into different types. For the same reason, it will be difficult for customers to explore the API. By the link, videos and presentations are available through auto-completion in the IDE (however, it may be for the better - although the documentation is read). Therefore, with such an interface, it is desirable to have detailed documentation and more examples of using the library.

UPD: the material presented in the article served as the basis for the report “Practical API for Data Warehouse” , which the author spoke on July 4 at the C ++ User Group in Nizhny Novgorod. The link is available video and presentation in pdf.