Scalable JSON serialization / deserialization library
Not so long ago, I participated in the project of writing firmware for some device. In the process of work, the question arose, and how, in fact, to interact with the "big brother" (computer manager)? Since completely different devices (various smartphones, tablets, laptops with different operating systems, etc.) were laid as “big brother”, it was planned to use a web application that dictated the use of JSON for messaging.
The result was an easy and fast JSON serialization / deserialization library. The main features of this library:
Initially, writing my own library serialized me to encourage this post. Unfortunately, that option did not suit me, because it uses STL and use it on the controller, in which there is only 1MB of flash and 198KB of RAM, to put it mildly, it is strange. But I liked the idea of ​​describing fields for serialization. Similar syntax for boost :: serialization looks similar. He was taken as a basis.
')
The serialization process is trivial - we are, in fact, bypassing the serializable data structure in depth. As you know, this action can be described recursively , due to which it is possible to refuse to allocate dynamic memory (use only the stack). It is clear that everything will collapse (more precisely, it will freeze) when I try to serialize a couple of objects that link to each other, but such cases have not come across to me yet.
Deserialization is a bit more complicated. In the process of thinking and after, getting to know someone else's creativity, I got the impression that there can be two types of deserializers (according to the initial data for building the parse tree):
Also parsers can be divided into:
Parsers building the parse tree on the incoming line have a slight advantage (in my opinion, the mythical one). Suppose we send a request to the server and wait for a response in this format:
But the answer comes like this:
In this situation, we can try to display the message fields on the expected structure and if it does not work out - and we know that we can get an error - try ("manually") to display the message on the structure with an error. Parsers building a parse tree on the transferred object are less flexible - they will simply discard the message as erroneous.
Why do I think this advantage is doubtful? Consider an example based on the digitalocean REST API .
Take, for example, the server part. When interacting with the server, the client accesses a specific URL by a specific method, passing JSON in the message body. For example:
Create a new Domain
To create a new domain, send a POST request to / v2 / domains. Set the "name" attribute to you. Set the "ip_address" attribute to the IP address.
URL - " api.digitalocean.com/v2/domains ".
Method - POST .
JSON message:
Any other message will be an error.
Also with the client side. If successful, the server responds with the status “201 Created” and a specific JSON message:
If an error occurs during the execution of the request, the status changes accordingly:
HTTP / 1.1 403 Forbidden
Thus, with proper construction of the protocol of interaction between the client and the server, no deserializer will have any problems.
Deserializers, however, who build the parse tree on the transferred object, have their advantage - under certain conditions they allow parsing the received string without using dynamic memory. In my opinion, this is an important advantage, since the dynamic memory is significantly slower compared to the stack, subject to leakage and fragmentation. The situation with the speed of memory allocation worsens even more in a multi-threaded environment.
And here we come to the main point. In the device under development, it was decided to use scmRTOS (well, I like it - it is small and it works), and there is no memory manager in it, which hinted at writing my own. And since all the rest of the logic was implemented without the use of dynamic memory and it is well known from the course of algorithms that context-free grammars (to which JSON should be treated) can be parsed with a stack machine, it was decided to try to write a deserializer using only the stack.
The serializer in the firmware was used very actively and (subjectively, since it was not compared with anything) showed excellent results. Unfortunately, by the time the device was finished working on the device, the deserializer had not yet been written. Nevertheless, the idea captured, and as a result, it was added, so on. "In production" has not yet been used.
The serializer itself consists of two classes: the JSONSerializer class and the Serializer class inheriting from it (which will allow us to further implement serialization in XML, at least I hope so). Actually, Serializer implements the tree traversal logic, and JSONSerializer converts data to text and transfers text to Handler for further sending to the counterparty.
Handler's interface looks like this:
The bool operator () operator (const char * str, uint32_t len) receives a message in portions as it is serialized. The call to bool SerializeEnd () reports that the object has been serialized. This was done for one simple reason: since the serializer knows nothing about where the final message is displayed (for example, in USB) and, accordingly, does not know whether the message will be fragmented during transmission or turn into additional fields — formation (with necessary), the filling and forwarding of the buffer was assigned to the Handler .
To serialize an object of a certain class, you need to inherit this class from the jsmincpp :: serialize :: Serialized class. This action is necessary to select the correct overloaded function inside the serializer. It does not carry any special load, since the jsmincpp :: serialize :: Serialized class does not contain fields.
It is also necessary to implement the function
For example, for some class it will look like this:
If it is necessary to serialize nested objects, their classes must be subjected to the same modifications.
Further serialization looks elementary:
If you, like me, have a serializer in a separate thread and take objects for serialization from its message queue, the previous version does not fit (why drag RTTI to the controller when we have already abandoned dynamic memory?). There is a notorious question with a camel. It will be solved by a certain base class, from which we will inherit our serializable class and put a pointer to it in the queue. This class is AbstractSerialized . It, in turn, is inherited from Serialized , which allows serializing the resulting class in the above scenario (if necessary).
It looks like this:
Behind this, we are done with the serialization of objects.
I absolutely do not like "things in themselves." If there is a choice, I prefer something that can be expanded (especially great if it is a living space) and adapted to suit my needs. This serializer can also be expanded by allowing arbitrary classes to be output to a channel in a specific way. Let me explain with an example what this means.
In the device under development, messaging with the “big brother” was carried out (as a first approximation) via USART at a speed of 19,200 bps. The longest message contained an array of 6 floats. Since the developed device used 6 absolute encoders with an accuracy of about 0.5 degrees and, accordingly, the absolute values ​​of the values ​​did not exceed 360 degrees, the serialized value for the float looked like this: 222.001999 . It has 5 extra digits (in fact, the last half of the characters are superfluous and do not carry a semantic load). You can somewhat speed up the exchange of messages, if you throw out extra characters. We cannot in any way influence the float's serialization library, but we can write a serializer for an arbitrary class. Thus, the class FloatPoint_3x1 was created.
The class itself looks like this:
Nothing special - a container for data. Note, you don't need to inherit it from Serialized !
The serialization function for it looks like this:
Everything is simple - we form a text string in the buffer and output it to the Handler . As a result, the serialized value looks like this: 222.0 .
This feature is a very powerful thing - we have full control over the output stream.
Deserialization is done by the Deserializer class (unexpected, right?), Which can work in two modes:
When instantiating a deserialization class, the first template parameter and the first parameter are passed to the constructor, the class responsible for reading the received data, with the following minimum set of methods:
The idea is peeped at the STL input iterators and working with it should look familiar.
The SymbolStream & operator ++ () method reads the next character from the input device (if read is buffered, it moves to the next character in the buffer) or gets up in anticipation of the arrival of a new character.
The char operator * () method returns the current character.
The SymbolStream End () and bool operator == (const SymbolStream & other) methods are designed to determine when the input stream has reached the end-of-stream state (end of stream).
When the deserializer class is instantiated by the second template parameter, a list of classes is transferred, into which fields the received messages can be deserialized.
Since telepaths, as usual, are on vacation, and the magic is not available to us, the deserializer must decide on the basis of some signs what message was specifically taken. By a titanic effort of will, it was decided to accept the message format as follows:
Nested classes are allowed, the nesting level is determined by the free space on the stack. This mode is suitable, for example, for embedded devices with data exchange via USB CDC ACM (virtual COM port) or USART, i.e., where there is one channel for exchanging all messages and there are no signs to determine which class belongs message. This somewhat limits the use of deserialization in ready-made solutions without additional adaptation, but is ideally suited for newly designed systems.
In the program it looks like this:
Where “Object” and “OtherObject” are the names of classes being deserialized (not instantiated objects! The deserializer will create objects itself).
The third template parameter and the second parameter of the constructor are passed to the allocator (although, probably, it is more correct to say the factory). This class has the following interface:
The T * Create (const T &) method allocates memory and allocates the created “T” object in it.
The void Delete (T *) method, respectively, deletes a specific object created earlier.
The deserialization itself is performed by the bool Deserialize (H & handler) method of the deserializer. A handler is passed to its input, which will receive control if the received message is successfully deserialized. Its interface is as follows:
Actually, one overloaded operator () per class from the list of messages being deserialized.
This is necessary because I cannot return different classes from the bool Deserialize (H & handler) method of the deserializer. Only in the Handler methods we have access to the type of the deserialized message. Knowing the type, the received message can, for example, be placed in the necessary queue for subsequent processing by another thread or processed on the spot.
In case of successful deserialization, the bool Deserialize (H & handler) method returns true and false in case of an error.
We deserialize one message at a time, and to continue processing, we call the method again (the necessary number of times).
Looks like this:
By default, the StaticCreator class is used as the Creator :
It creates the required objects in its buffer.
Thus, after deserialization, in the Handler handler, you must copy the deserialized object somewhere. In the prototype of this serializer, I passed the object by value to the queue of another thread for further processing (which made it possible to completely abandon the use of dynamic memory). If this behavior does not suit and there is an opportunity to use dynamic memory, you need to use another Creator or write your own.
MallocCreator is available in the library , using Malloc () to allocate objects and SharedPrtCreator , which allows the use of smart pointers ( std :: shared_ptr ). For more, I did not have enough imagination.
If we know the specific type of message being deserialized, we can use the second mode of the deserializer. Just pass a pointer to the object into which we want to deserialize the received message to the overloaded bool Deserialize method (O * obj) .
You can specify the NullObj dummy object as the only member of the deserialization list. It looks like this:
Or in the case of using shared_ptr , like this:
In this mode of operation (regardless of the transferred Creator ), the dynamic memory is not used by the deserializer ( Creator is instantiated, but its methods are not called).
This deserializer allows the extension of its functionality. First, the possibility of expanding the number of user classes being deserialized. Secondly, the ability to change the memory allocation strategy for the class being deserialized.
Consider the extension of the functional on the example of some class StaticString . This class will allow us to deserialize strings in systems where there is no dynamic memory. Of course, it contains many restrictions, but with some skill you can use it. The class looks like this:
Actually, nothing special - a static array for characters, around which the logic of working with a string is built.
Deserialization is done with the following code:
As you can see, we have full access to the input stream of characters, therefore, we can deserialize anything.
At the stage of writing the concept, the following problem arose: JSON itself does not limit the length of the parameter name, which requires, generally speaking, the need to use dynamic memory (with the possibility of increasing the size of the allocated buffer), or a static buffer of sufficient size to accommodate the longest name. It was a pity for the memory (development, I recall, was originally conducted for the microcontroller), so the following idea was realized: what if not to accumulate the characters of the parameter name in the buffer and then compare it with the names of the deserialized parameters, and take the CRC32 from the input string, and later compare with the calculated at the compilation stage (constexpr function) CRC32 from the field names of the deserializable class. This saves us memory (instead of strings, only uint32_t is stored) and speeds up the comparison, but adds headaches to possible CRC32 collisions from parameter names. What can I say ... Test your code more, the tests should catch such problems! You are testing your code, right?
In the process of reading the forums, sometimes I came across messages from the sufferers asking them to indicate the JSON serializer / deserializer with STL support. There was an overwhelming desire to support the afflicted. Which was implemented - like all the main STL containers are supported, both by the serializer and by the deserializer. If something is not supported, you can always finish the support. There is nothing more to say about this.
The serializer itself is rather trivial, so there is no desire even to compare it with anyone.
It was more interesting to compare the deserializer with competing libraries. Since I didn’t find C ++ libraries suitable for use on the controller (I didn’t look for it in general), but adding STL support and expandability transferred the product to another consumer category — the ability to use on full-fledged servers, I compared the libraries with those libraries that I recommend using found on the forums. The comparison, of course, is not comprehensive - only four competitors. But I did not want to test competitors further, because the results, in my opinion, were very depressing. The tests themselves are here , in the subdirectories of the / src directory in the * .mk files, correct the paths to the compiler and libraries (except Qt. To build it, create a project in QtCreator and copy * .cpp into it). The build is done in the root directory by calling make <benchmark>. You can view the build projects simply by calling make.
So, the following libraries were tested (besides the one developed):
The comparison was made as follows: according to the documentation of a specific library (well, naturally, since I understood it), an application was written, the task of which was to deserialize some (syntactically and semantically correct) lines into an object of a certain class. And so 1 000 000 times, because on my laptop (i3) fewer iterations took place in a very short time.Work time was measured by the time command and was taken from the “user: XXX” line. It is clear that the test does not pretend to be serious, but some conclusions can be made. Tests were run 100 times. The results are presented in the table.
Some comments.
But seriously, can anyone know the fast JSON deserialization libraries? It would be interesting to compare the possibilities and look at the internal structure.
The result was an easy and fast JSON serialization / deserialization library. The main features of this library:
- in the basic functionality (without using containers STL) does not use dynamic memory, in general;
- consists only of header files (headers-only);
- there is support for STL containers;
- allows you to create extensions to handle arbitrary types.
Some lyrics
Initially, writing my own library serialized me to encourage this post. Unfortunately, that option did not suit me, because it uses STL and use it on the controller, in which there is only 1MB of flash and 198KB of RAM, to put it mildly, it is strange. But I liked the idea of ​​describing fields for serialization. Similar syntax for boost :: serialization looks similar. He was taken as a basis.
')
The serialization process is trivial - we are, in fact, bypassing the serializable data structure in depth. As you know, this action can be described recursively , due to which it is possible to refuse to allocate dynamic memory (use only the stack). It is clear that everything will collapse (more precisely, it will freeze) when I try to serialize a couple of objects that link to each other, but such cases have not come across to me yet.
Deserialization is a bit more complicated. In the process of thinking and after, getting to know someone else's creativity, I got the impression that there can be two types of deserializers (according to the initial data for building the parse tree):
- build a parse tree on the input line, then try on this tree to the object into which the message is deserialized (for example, Qt, jsoncpp or jsmn);
- they build a parse tree on the transferred object, and then an accepted string is tried on (parser from cxxtools and the proposed library).
Also parsers can be divided into:
- parsers that require a string containing a JSON message (Qt, jsoncpp, and jsmn) for their work;
- "Online" parsers, processing individual message symbols (cxxtools and the proposed library).
Parsers building the parse tree on the incoming line have a slight advantage (in my opinion, the mythical one). Suppose we send a request to the server and wait for a response in this format:
{ one : 10.01, two : 20.02 }, But the answer comes like this:
{ error : <- > }. In this situation, we can try to display the message fields on the expected structure and if it does not work out - and we know that we can get an error - try ("manually") to display the message on the structure with an error. Parsers building a parse tree on the transferred object are less flexible - they will simply discard the message as erroneous.
Why do I think this advantage is doubtful? Consider an example based on the digitalocean REST API .
Take, for example, the server part. When interacting with the server, the client accesses a specific URL by a specific method, passing JSON in the message body. For example:
Create a new Domain
To create a new domain, send a POST request to / v2 / domains. Set the "name" attribute to you. Set the "ip_address" attribute to the IP address.
URL - " api.digitalocean.com/v2/domains ".
Method - POST .
JSON message:
{"name":"example.com","ip_address":"1.2.3.4"}. Any other message will be an error.
Also with the client side. If successful, the server responds with the status “201 Created” and a specific JSON message:
{ "domain": { "name": "example.com", "ttl": 1800, "zone_file": null } }. If an error occurs during the execution of the request, the status changes accordingly:
HTTP / 1.1 403 Forbidden
{ "id": "forbidden", "message": "You do not have access for the attempted action." }. Thus, with proper construction of the protocol of interaction between the client and the server, no deserializer will have any problems.
Deserializers, however, who build the parse tree on the transferred object, have their advantage - under certain conditions they allow parsing the received string without using dynamic memory. In my opinion, this is an important advantage, since the dynamic memory is significantly slower compared to the stack, subject to leakage and fragmentation. The situation with the speed of memory allocation worsens even more in a multi-threaded environment.
And here we come to the main point. In the device under development, it was decided to use scmRTOS (well, I like it - it is small and it works), and there is no memory manager in it, which hinted at writing my own. And since all the rest of the logic was implemented without the use of dynamic memory and it is well known from the course of algorithms that context-free grammars (to which JSON should be treated) can be parsed with a stack machine, it was decided to try to write a deserializer using only the stack.
The serializer in the firmware was used very actively and (subjectively, since it was not compared with anything) showed excellent results. Unfortunately, by the time the device was finished working on the device, the deserializer had not yet been written. Nevertheless, the idea captured, and as a result, it was added, so on. "In production" has not yet been used.
Serializer
The serializer itself consists of two classes: the JSONSerializer class and the Serializer class inheriting from it (which will allow us to further implement serialization in XML, at least I hope so). Actually, Serializer implements the tree traversal logic, and JSONSerializer converts data to text and transfers text to Handler for further sending to the counterparty.
Handler's interface looks like this:
struct SerializeHandler { bool operator()( const char *str, uint32_t len ); bool SerializeEnd( ); }; The bool operator () operator (const char * str, uint32_t len) receives a message in portions as it is serialized. The call to bool SerializeEnd () reports that the object has been serialized. This was done for one simple reason: since the serializer knows nothing about where the final message is displayed (for example, in USB) and, accordingly, does not know whether the message will be fragmented during transmission or turn into additional fields — formation (with necessary), the filling and forwarding of the buffer was assigned to the Handler .
Serialization of a specific class
To serialize an object of a certain class, you need to inherit this class from the jsmincpp :: serialize :: Serialized class. This action is necessary to select the correct overloaded function inside the serializer. It does not carry any special load, since the jsmincpp :: serialize :: Serialized class does not contain fields.
It is also necessary to implement the function
bool Serialize(Serializer &) const { … } For example, for some class it will look like this:
struct SerializedClass : public Serialized { int8_t One; uint8_t Two; SerializedClass( ) : One( 0 ), Two( 0 ) { } SerializedClass( int8_t one, uint8_t two ) : One( one ), Two( two ) { } template < typename S > bool Serialize( S &serializer ) const { SERIALIZE( One ); SERIALIZE( Two ); return true; } }; If it is necessary to serialize nested objects, their classes must be subjected to the same modifications.
Further serialization looks elementary:
typedef Serializer < JSONSerializer < SerializeHandler > > Serializer_t; SerializeHandler handler; Serializer_t serializer( handler ); SerializedClass obj; … serializer.Serialize( obj ) Serialization from a pointer to the base class or how to drag a camel into the eye of a needle
If you, like me, have a serializer in a separate thread and take objects for serialization from its message queue, the previous version does not fit (why drag RTTI to the controller when we have already abandoned dynamic memory?). There is a notorious question with a camel. It will be solved by a certain base class, from which we will inherit our serializable class and put a pointer to it in the queue. This class is AbstractSerialized . It, in turn, is inherited from Serialized , which allows serializing the resulting class in the above scenario (if necessary).
It looks like this:
typedef Serializer < JSONSerializer < SerializeHandler > > Serializer_t; class SerializeObj : public AbstractSerialized < Serializer_t > { … virtual bool Serialize( Serializer_t &serializer ) const override { … } }; … SerializeObj *obj = new SerializeObj; … queue.Send( obj ); … SerializeHandler handler; Serializer_t serializer( handler ); AbstractSerialized < Serializer_t > *obj = queue.Get( ); serializer.Serialize( obj ); … Behind this, we are done with the serialization of objects.
Extension of the serializer functionality
I absolutely do not like "things in themselves." If there is a choice, I prefer something that can be expanded (especially great if it is a living space) and adapted to suit my needs. This serializer can also be expanded by allowing arbitrary classes to be output to a channel in a specific way. Let me explain with an example what this means.
In the device under development, messaging with the “big brother” was carried out (as a first approximation) via USART at a speed of 19,200 bps. The longest message contained an array of 6 floats. Since the developed device used 6 absolute encoders with an accuracy of about 0.5 degrees and, accordingly, the absolute values ​​of the values ​​did not exceed 360 degrees, the serialized value for the float looked like this: 222.001999 . It has 5 extra digits (in fact, the last half of the characters are superfluous and do not carry a semantic load). You can somewhat speed up the exchange of messages, if you throw out extra characters. We cannot in any way influence the float's serialization library, but we can write a serializer for an arbitrary class. Thus, the class FloatPoint_3x1 was created.
The class itself looks like this:
class FloatPoint_3x1 { public: FloatPoint_3x1( ) : _val( 0.0f ) { } FloatPoint_3x1( float val ) : _val( val ) { } float GetValue( ) const { return _val; } private: float _val; }; Nothing special - a container for data. Note, you don't need to inherit it from Serialized !
The serialization function for it looks like this:
template < typename S > bool operator <<( S &serializer, const FloatPoint_3x1 &data ) { const char f [ ] = "%3.1f"; char buffer [ 10 ]; uint32_t len = ::sprintf( buffer, f, data.GetValue( ) ); if( len > 0 ) return serializer.GetHandler( )( buffer, len ); return false; } Everything is simple - we form a text string in the buffer and output it to the Handler . As a result, the serialized value looks like this: 222.0 .
This feature is a very powerful thing - we have full control over the output stream.
Deserializer
Deserialization is done by the Deserializer class (unexpected, right?), Which can work in two modes:
- gets when creating a list of classes into which it will try to deserialize received messages;
- receives as input an object of a specific class, into the fields of which it will try to deserialize the received message.
When instantiating a deserialization class, the first template parameter and the first parameter are passed to the constructor, the class responsible for reading the received data, with the following minimum set of methods:
class InputStream { public: SymbolStream & operator++( ); char operator*( ); bool operator==( const SymbolStream &other ); SymbolStream End( ); }; The idea is peeped at the STL input iterators and working with it should look familiar.
The SymbolStream & operator ++ () method reads the next character from the input device (if read is buffered, it moves to the next character in the buffer) or gets up in anticipation of the arrival of a new character.
The char operator * () method returns the current character.
The SymbolStream End () and bool operator == (const SymbolStream & other) methods are designed to determine when the input stream has reached the end-of-stream state (end of stream).
Deserialization with automatic selection from the list of classes
When the deserializer class is instantiated by the second template parameter, a list of classes is transferred, into which fields the received messages can be deserialized.
Since telepaths, as usual, are on vacation, and the magic is not available to us, the deserializer must decide on the basis of some signs what message was specifically taken. By a titanic effort of will, it was decided to accept the message format as follows:
{"< >":{< >}}. Nested classes are allowed, the nesting level is determined by the free space on the stack. This mode is suitable, for example, for embedded devices with data exchange via USB CDC ACM (virtual COM port) or USART, i.e., where there is one channel for exchanging all messages and there are no signs to determine which class belongs message. This somewhat limits the use of deserialization in ready-made solutions without additional adaptation, but is ideally suited for newly designed systems.
In the program it looks like this:
class Object; class OtherObject; typedef ObjectsList < DESERIALIZEOBJ( Object ), DESERIALIZEOBJ( OtherObject ) > SerializeList_t; Where “Object” and “OtherObject” are the names of classes being deserialized (not instantiated objects! The deserializer will create objects itself).
The third template parameter and the second parameter of the constructor are passed to the allocator (although, probably, it is more correct to say the factory). This class has the following interface:
class Creator { public: template < typename T > T * Create( const T & ); template < typename T > void Delete( T * ); }; The T * Create (const T &) method allocates memory and allocates the created “T” object in it.
The void Delete (T *) method, respectively, deletes a specific object created earlier.
The deserialization itself is performed by the bool Deserialize (H & handler) method of the deserializer. A handler is passed to its input, which will receive control if the received message is successfully deserialized. Its interface is as follows:
class Handler { public: bool operator()( Object *param ); bool operator()( OtherObject *param ); … }; Actually, one overloaded operator () per class from the list of messages being deserialized.
This is necessary because I cannot return different classes from the bool Deserialize (H & handler) method of the deserializer. Only in the Handler methods we have access to the type of the deserialized message. Knowing the type, the received message can, for example, be placed in the necessary queue for subsequent processing by another thread or processed on the spot.
In case of successful deserialization, the bool Deserialize (H & handler) method returns true and false in case of an error.
We deserialize one message at a time, and to continue processing, we call the method again (the necessary number of times).
Looks like this:
class Object; class OtherObject; … typedef ObjectsList < DESERIALIZEOBJ( Object ), DESERIALIZEOBJ( OtherObject ) > SerializeList_t; … SocketStream s( socket ); DeserializeHandler h( ); Deserializer < SymbolStream, SerializeList_t > d( s ); if( !d.Deserialize( h ) ) DeserializeErrorHandler( ); A few words about the non-use of dynamic memory
By default, the StaticCreator class is used as the Creator :
template < uint32_t BuffSize > class StaticCreator { public: uint32_t _buff [ BuffSize / 4 + 1 ]; template < typename T > T * Create( const T & ) { return new ( _buff ) T; } template < typename T > void Delete( T * ) { } }; It creates the required objects in its buffer.
Thus, after deserialization, in the Handler handler, you must copy the deserialized object somewhere. In the prototype of this serializer, I passed the object by value to the queue of another thread for further processing (which made it possible to completely abandon the use of dynamic memory). If this behavior does not suit and there is an opportunity to use dynamic memory, you need to use another Creator or write your own.
MallocCreator is available in the library , using Malloc () to allocate objects and SharedPrtCreator , which allows the use of smart pointers ( std :: shared_ptr ). For more, I did not have enough imagination.
Deserialization of a specific class
If we know the specific type of message being deserialized, we can use the second mode of the deserializer. Just pass a pointer to the object into which we want to deserialize the received message to the overloaded bool Deserialize method (O * obj) .
You can specify the NullObj dummy object as the only member of the deserialization list. It looks like this:
SocketStream s( socket ); typedef ObjectsList < DESERIALIZEOBJ(NullObj) > SerializeList_t; Deserializer < SymbolStream, SerializeList_t > d( s ); Object obj; d.Deserialize( &obj ); Or in the case of using shared_ptr , like this:
SocketStream s( socket ); typedef ObjectsList < DESERIALIZEOBJ(NullObj) > SerializeList_t; Deserializer < SymbolStream, SerializeList_t > d( s ); auto obj = make_shared< Object >(); d.Deserialize( obj.get( ) ); In this mode of operation (regardless of the transferred Creator ), the dynamic memory is not used by the deserializer ( Creator is instantiated, but its methods are not called).
Extension of deserialization functionality
This deserializer allows the extension of its functionality. First, the possibility of expanding the number of user classes being deserialized. Secondly, the ability to change the memory allocation strategy for the class being deserialized.
Consider the extension of the functional on the example of some class StaticString . This class will allow us to deserialize strings in systems where there is no dynamic memory. Of course, it contains many restrictions, but with some skill you can use it. The class looks like this:
template < uint32_t Num > class StaticString { public: StaticString( ) : _length( 0 ) { _buff [ 0 ] = 0; } StaticString( const char *str ) { Assign( str ); } bool Add( char symbol ) { if( Num == _length ) return false; _buff[ _length++ ] = symbol; _buff[ _length ] = 0; return true; } bool Add( const char *str ) { uint32_t strSize = ::strlen( str ); if( strSize > Num - _length ) return false; ::strcpy( _buff, str ); _length += strSize; _buff[ _length + 1 ] = 0; return true; } bool Assign( const char *str ) { _length = 0; _buff[ _length ] = 0; return Add( str ); } const char * GetString( ) { return _buff; } uint32_t GetLength( ) { return _length; } private: uint32_t _length; char _buff [ Num + 1 ]; }; Actually, nothing special - a static array for characters, around which the logic of working with a string is built.
Deserialization is done with the following code:
template < uint32_t Hash, uint32_t Num > class StaticStringParam { public: enum { HASH = Hash }; StaticStringParam( StaticString< Num > ¶m ) : _param( param ) { } template < typename D > bool Parse( D &deserializer ) { return ParseStaticString( _param, deserializer.GetStream( ) ); } private: StaticString< Num > &_param; }; template < uint32_t Hash, uint32_t Num > StaticStringParam < Hash, Num > MakeParam( StaticString< Num > ¶m ) { return StaticStringParam < Hash, Num >( param ); } As you can see, we have full access to the input stream of characters, therefore, we can deserialize anything.
Some limitations when using this deserializer
At the stage of writing the concept, the following problem arose: JSON itself does not limit the length of the parameter name, which requires, generally speaking, the need to use dynamic memory (with the possibility of increasing the size of the allocated buffer), or a static buffer of sufficient size to accommodate the longest name. It was a pity for the memory (development, I recall, was originally conducted for the microcontroller), so the following idea was realized: what if not to accumulate the characters of the parameter name in the buffer and then compare it with the names of the deserialized parameters, and take the CRC32 from the input string, and later compare with the calculated at the compilation stage (constexpr function) CRC32 from the field names of the deserializable class. This saves us memory (instead of strings, only uint32_t is stored) and speeds up the comparison, but adds headaches to possible CRC32 collisions from parameter names. What can I say ... Test your code more, the tests should catch such problems! You are testing your code, right?
STL container support
In the process of reading the forums, sometimes I came across messages from the sufferers asking them to indicate the JSON serializer / deserializer with STL support. There was an overwhelming desire to support the afflicted. Which was implemented - like all the main STL containers are supported, both by the serializer and by the deserializer. If something is not supported, you can always finish the support. There is nothing more to say about this.
Comparison and benchmarks
The serializer itself is rather trivial, so there is no desire even to compare it with anyone.
It was more interesting to compare the deserializer with competing libraries. Since I didn’t find C ++ libraries suitable for use on the controller (I didn’t look for it in general), but adding STL support and expandability transferred the product to another consumer category — the ability to use on full-fledged servers, I compared the libraries with those libraries that I recommend using found on the forums. The comparison, of course, is not comprehensive - only four competitors. But I did not want to test competitors further, because the results, in my opinion, were very depressing. The tests themselves are here , in the subdirectories of the / src directory in the * .mk files, correct the paths to the compiler and libraries (except Qt. To build it, create a project in QtCreator and copy * .cpp into it). The build is done in the root directory by calling make <benchmark>. You can view the build projects simply by calling make.
So, the following libraries were tested (besides the one developed):
- jsmn is a C project, but does not use dynamic memory, it was interesting to compare;
- Qt ;
- jsoncpp ;
- cxxtools .
The comparison was made as follows: according to the documentation of a specific library (well, naturally, since I understood it), an application was written, the task of which was to deserialize some (syntactically and semantically correct) lines into an object of a certain class. And so 1 000 000 times, because on my laptop (i3) fewer iterations took place in a very short time.Work time was measured by the time command and was taken from the “user: XXX” line. It is clear that the test does not pretend to be serious, but some conclusions can be made. Tests were run 100 times. The results are presented in the table.
| Library / framework | Best execution time, with | Relation to the leader of the test | Average execution time, with | Relation to the leader of the test |
| jsmincpp | 0.219 | - | 0.22252 | - |
| jsmn | 0.595 | 2.716895 | 0.60017 | 2.697151 |
| Qt | 1.359 | 6.205479 | 1.54353 | 6.93659 |
| jsoncpp | 4.981 | 22.74429 | 5.89901 | 26.51002 |
| cxxtools | 5.26 | 24.01826 | 5.95608 | 26.76649 |
Some comments.
- In terms of execution speed, we almost made a program with a library in C (who claimed that C ++ is slow? If you are writing in C ++, is it not worth thinking about your professional suitability?).
- , , . , , cxxtools shared 6.391 , 7.63804 , . . shared . : Qt . , shared , , Qt — 0.4 — 0.6 . ( , C++ ?!).
- striped :
- jsmincpp – 8040 ,
- jsmn – 8792 .
700 ( 10%) C' . ( , C++ ? , !) - jsmincpp – 8040 ,
But seriously, can anyone know the fast JSON deserialization libraries? It would be interesting to compare the possibilities and look at the internal structure.
Source: https://habr.com/ru/post/268015/
All Articles