Serialization of objects in json format for implementing REST API
Already, the version of Symfony 2.1 will be released, and in the community it is still impossible to implement a full-featured REST "without crutches", and, in my opinion, there is something wrong here. Recently there was an article with the big name REST API's with Symfony2: The Right Way , but, in essence, it only confirms my words. All problem rests against serialization and deserialization of objects. It would seem that the simplest task and solutions should be many, but, unfortunately, no. Let's do everything in order.
JMSSerializerBundle, perhaps, is a trendsetter in serialization at the moment (he is also advertised by the FOSRestBundle). It is multifunctional; it can store the rules of serialization in various formats, serialize and deserialize data into various formats, uses a cache. But he has several small unsolved problems, they have been touched many times, and their solution is not foreseen. It seems to me that, in pursuit of multi-functionality, the application architecture has reached a dead end.
The first problem is a bundle that pulls a bunch of dependencies, accordingly, it cannot be used outside of Symfony 2. This is very strange, because the question initially concerned serialization, and it is not clear why not to make it a library.
')
The second problem is the impossibility of serializing null values. With reference to our REST API, this is unacceptable. There is a very big issue , but I am more than sure that there will be no solution. In fact, the situation is very strange. If you take json_decode and json_encode, they correctly handle this task. For the xml format, I solved this problem a year ago with the implementation of doctrine-oxm .
The third problem is the inability to deserialize into an existing data object. This makes it impossible to use POST / PATCH requests that produce partial data updates. The main question is how all this time FOSRestBundle “fools” people about the easy implementation of REST in Symfony 2.
We could not believe for a long time that no one ever made a full-fledged implementation of REST in Symfony2 and did not encounter these problems. We looked at a bunch of solutions, and, as the immortal Choi sang, “it's all wrong, and everything is wrong.” One day, we were pleased with Benjamin Eberlei , who drew attention to a third problem, creating his own SimpleThingsFormSerializerBundle bundle. But, unfortunately, like the entire Symfony2 community, it “went crazy” on version 2.1, which even had no beta version. But this is another story, and, to our happiness, there was a person who made compatibility with 2.0.
So it seems that happiness is at hand, and we can update our models. No matter how hard it is, we are ready to go against our will and create a FormType for our DTO (Data Transfer Object) objects, completely replacing JMSSerializerBundle with SimpleThingsFormSerializerBundle. They did it, but they did not find happiness. As it turned out, the new bundle converts all information into strings, i.e. our client will never see either numeric or boolean values. We did not receive an answer to the question why this was done, but there was a suggestion to use JMSSerializer for serialization, and FormSerializer for deserialization. But it seems to me that something is wrong here. In addition, the symfony form in version 2.0 can produce bind only with GET or POST requests, ignoring the rest. Yes, and I have a lot of "fi", about the use of forms for REST, leave it outside the article. I went on vacation for two weeks with the hope that something will change. But…
The task seems simple; in our REST API, we are guaranteed to provide information in json format. Other formats do not interest us (it seems to me, like most modern projects). We assume that we have DTO objects that always have set / get methods for their attributes. We want to convert these DTOs to json and back (with solving the problems described above). We really like the ability to store the serialization rules in the yml format and from the variety of settings of the JMSSerializerBundle we only need to set the field mapping (“serialized_name” and “type”) and the “expose” flag.
Before proceeding with the implementation of my library, I once again searched for current solutions. Two interesting projects were found:
FbsSerializer - it seems to me, the “predecessor” of JMSSerializerBundle. The possibilities and the underlying ideas for implementation are very similar, the same problems are present, and also the collections are serialized in an inappropriate way;
ObjectSerializer is a very simple implementation, almost one to one coincided with the way I saw my library for myself. The main disadvantage is that class mapping is passed through the constructor, which is unacceptable for us.
Recently, I have become a fan of ease of implementation. The simpler the code, the less it takes responsibility - the better it is. I do not want to write code in which functionality is laid, which no one will ever need, while the architecture of the application is broken.
The basic idea is to convert a complex object with different serialization rules into an array. This is what the ArrayAdapter does, and it encapsulates the entire data conversion logic. The generated array is easily converted to json format, I think there are no difficulties for xml. During deserialization, the opposite is true: we convert the input data into an array, then the ArrayAdapter converts the data into an object according to its own rules.
In our case, serializerAdapter represents a simple wrapper over json_encode, json_decode functions. All logic that interests us is in the ArrayAdapter. Here we find the ideological difference in the processing of configuration data and the serialization of this library and the FbsSerializer, JMSSerializerBundle.
In my view, we have some set of rules for serializing objects, located in some files in a certain space. First we need to get our configuration from this space. But the formats in which the configuration is stored can be many, and it is logical to transform this into an object representation for ease of use. Johannes Schmitt does this with the metadata library. She is very good, but closely related to the Reflection object. Therefore, later, during serialization, all work comes from the object being serialized and its Reflection. In a nutshell, we iterate over everything that is possible in an object, and we look at what settings we have for this attribute or method. I believe that this approach to intertwining with complex processing of values ​​(it could have been made easier) is one of the reasons for the unresolved problems. And in my implementation I do the opposite. We simply convert the configuration into an object representation that knows nothing about the object being serialized. Next, we iterate over our metadata and see if we can serialize the current attribute, what type it is and how to process it, under what name we serialize it. During data processing, we call the getter / setter we need. Thus, we proceed not from an object, but from a given configuration. As they say, they asked for it and got it. Below is a snippet of code:
To handle the values ​​for serialization / deserialization, the private method handleValue (about 60 lines) is currently used.
Initially, to convert settings from a yml file to an object, I wanted to write something of my own, since the task is very simple. You need to abstract from two things: the location of the configuration files (they can be stored anywhere: the file system, memory, database) and their format (yml, json, ini, xml). At the same time, as the Thimlide noted, “we will not parse the * .yml file every time”, so we need a cache. Its interface is known to all: put, get, remove. But I stopped in time and remembered the metadata library, and again rested on Reflection that I did not need. Therefore, with minor corrections, I took this part of the code from there, saluting Johannes.
As a result, I was lost for a team of 33 hours, but the simple serializer library and the OpensoftSimpleSerializerBundle bundle were written .
Library dependencies:
Symfony / yaml component
Requirements:
- for the object being serialized, the rules of serialization should be described;
- the object to be serialized must have setters and getters for attributes.
Advantages compared to JMSSerializerBundle:
- does not have a number of urgent problems, such as: processing attributes with a null value, deserializing data into an existing object;
- setting the date formatting in the configuration;
- does not have a binding to the symfony 2 framework.
Naturally, this is not ideal. JMSSerializerBundle is more functional and has less restrictions. But, in our case, there is a clear bias to use when implementing the REST API. When we deal with REST in large applications, we will not be able to do without implementing the DTO pattern, since never will our models be explicitly reflected in the API. And, accordingly, the Assembler will most likely use the setter / getter methods when converting DTO to DomainObject. And the second requirement disappears by itself. The first, I think, is also doing everything. After all, by providing API to customers, we tell them what and in what format we accept or give. And it seems illogical if we will be missing these rules. As for the response in different formats, in our project we are dealing with json. It is very simple and convenient. XML, in my opinion, is already a bit old for use in REST. The HTML format offered by the FOSRestBundle is too contrived, I cannot imagine its use. For the reliability of the REST API meet behat tests (hello and thanks to Konstantin aka everzet ), I hope davert will not be offended, since the codeception is really cool.
It is worth talking separately about the possibility of configuring serialization. Options are minimized, although it is not difficult to add them. The file with configs is generally similar to JMSSerializerBundle:
The “expose” option is “false” by default, so for all attributes that we want to serialize, it must be specified. This is my personal preference, which is not allowed - something is forbidden. Perhaps affected the enthusiasm of the ACL in his youth. In principle, you can embed a full-fledged exlude / expose, if you wish. “Serialized_name” is an optional parameter; if it is not specified, the attribute will be serialized according to its name. The last option “type” is also optional. If it is not specified, then no actions with the attribute value will be performed. Otherwise, it will be cast to the specified type. Possible types: integer, boolean, double, string, array, T, array <T>, DateTime, DateTime <format>.
The main differences from the JMSSerializerBundle are the absence of an ArrayCollection and the presence of DateTime <format>. The first is missing fundamentally, since its presence adds the dependence of the library on Doctrine / Orm. If you are using the symfony 2 framework, then at 90% it will be satisfied. And what about the other 10 percent, or those who do not use symfony? Moreover, if we are talking about applying to the DTO, then there almost certainly will not be any collections, there is no point in them. But this problem is solved in the future. The second feature is the ability to specify how to format the date. I remind you that the DateTime entry assumes that you have a DateTime object. If I'm not mistaken, in JMSSerializerBundle you can create a custom handler for the date and specify the format in it. Everything is easier here; directly in the configuration, you can specify a DateTime class constant, for example, “ISO8601”, or a string with the necessary time formatting.
You can create a TODO list, although, to be honest, I do not see any urgent need for this:
1) add the ability to store the serialization configuration in other formats (annotation, xml);
2) add the ability to serialize data in formats other than json;
3) add a variety of options to the configuration;
4) to refactor the code in handleValue, incorporating the Visitor pattern, or Chain of Responsobility, or something else, thereby adding the ability to create a custom value handler;
5) something else.
Acknowledgments: I especially want to thank my team leader for allowing me to “withdraw into myself” and for my patience during the difficult pre-release time for the project, as well as the team that destroyed the bugs while I was high :). Thanks to Johannes, Benjamin - you are really cool guys, and we don’t like to reinvent the wheel, but we had no way out.
JMSSerializerBundle, perhaps, is a trendsetter in serialization at the moment (he is also advertised by the FOSRestBundle). It is multifunctional; it can store the rules of serialization in various formats, serialize and deserialize data into various formats, uses a cache. But he has several small unsolved problems, they have been touched many times, and their solution is not foreseen. It seems to me that, in pursuit of multi-functionality, the application architecture has reached a dead end.
The first problem is a bundle that pulls a bunch of dependencies, accordingly, it cannot be used outside of Symfony 2. This is very strange, because the question initially concerned serialization, and it is not clear why not to make it a library.
')
The second problem is the impossibility of serializing null values. With reference to our REST API, this is unacceptable. There is a very big issue , but I am more than sure that there will be no solution. In fact, the situation is very strange. If you take json_decode and json_encode, they correctly handle this task. For the xml format, I solved this problem a year ago with the implementation of doctrine-oxm .
The third problem is the inability to deserialize into an existing data object. This makes it impossible to use POST / PATCH requests that produce partial data updates. The main question is how all this time FOSRestBundle “fools” people about the easy implementation of REST in Symfony 2.
We could not believe for a long time that no one ever made a full-fledged implementation of REST in Symfony2 and did not encounter these problems. We looked at a bunch of solutions, and, as the immortal Choi sang, “it's all wrong, and everything is wrong.” One day, we were pleased with Benjamin Eberlei , who drew attention to a third problem, creating his own SimpleThingsFormSerializerBundle bundle. But, unfortunately, like the entire Symfony2 community, it “went crazy” on version 2.1, which even had no beta version. But this is another story, and, to our happiness, there was a person who made compatibility with 2.0.
So it seems that happiness is at hand, and we can update our models. No matter how hard it is, we are ready to go against our will and create a FormType for our DTO (Data Transfer Object) objects, completely replacing JMSSerializerBundle with SimpleThingsFormSerializerBundle. They did it, but they did not find happiness. As it turned out, the new bundle converts all information into strings, i.e. our client will never see either numeric or boolean values. We did not receive an answer to the question why this was done, but there was a suggestion to use JMSSerializer for serialization, and FormSerializer for deserialization. But it seems to me that something is wrong here. In addition, the symfony form in version 2.0 can produce bind only with GET or POST requests, ignoring the rest. Yes, and I have a lot of "fi", about the use of forms for REST, leave it outside the article. I went on vacation for two weeks with the hope that something will change. But…
The task seems simple; in our REST API, we are guaranteed to provide information in json format. Other formats do not interest us (it seems to me, like most modern projects). We assume that we have DTO objects that always have set / get methods for their attributes. We want to convert these DTOs to json and back (with solving the problems described above). We really like the ability to store the serialization rules in the yml format and from the variety of settings of the JMSSerializerBundle we only need to set the field mapping (“serialized_name” and “type”) and the “expose” flag.
Before proceeding with the implementation of my library, I once again searched for current solutions. Two interesting projects were found:
FbsSerializer - it seems to me, the “predecessor” of JMSSerializerBundle. The possibilities and the underlying ideas for implementation are very similar, the same problems are present, and also the collections are serialized in an inappropriate way;
ObjectSerializer is a very simple implementation, almost one to one coincided with the way I saw my library for myself. The main disadvantage is that class mapping is passed through the constructor, which is unacceptable for us.
Recently, I have become a fan of ease of implementation. The simpler the code, the less it takes responsibility - the better it is. I do not want to write code in which functionality is laid, which no one will ever need, while the architecture of the application is broken.
<?php class Serializer { public function serialize($object) { $array = $this->arrayAdapter->toArray($object); return $this->adapter->serialize($array); } public function unserialize($data, $object) { $array = $this->serializerAdapter->unserialize($data); return $this->arrayAdapter->toObject($array, $object); } } ?> The basic idea is to convert a complex object with different serialization rules into an array. This is what the ArrayAdapter does, and it encapsulates the entire data conversion logic. The generated array is easily converted to json format, I think there are no difficulties for xml. During deserialization, the opposite is true: we convert the input data into an array, then the ArrayAdapter converts the data into an object according to its own rules.
In our case, serializerAdapter represents a simple wrapper over json_encode, json_decode functions. All logic that interests us is in the ArrayAdapter. Here we find the ideological difference in the processing of configuration data and the serialization of this library and the FbsSerializer, JMSSerializerBundle.
In my view, we have some set of rules for serializing objects, located in some files in a certain space. First we need to get our configuration from this space. But the formats in which the configuration is stored can be many, and it is logical to transform this into an object representation for ease of use. Johannes Schmitt does this with the metadata library. She is very good, but closely related to the Reflection object. Therefore, later, during serialization, all work comes from the object being serialized and its Reflection. In a nutshell, we iterate over everything that is possible in an object, and we look at what settings we have for this attribute or method. I believe that this approach to intertwining with complex processing of values ​​(it could have been made easier) is one of the reasons for the unresolved problems. And in my implementation I do the opposite. We simply convert the configuration into an object representation that knows nothing about the object being serialized. Next, we iterate over our metadata and see if we can serialize the current attribute, what type it is and how to process it, under what name we serialize it. During data processing, we call the getter / setter we need. Thus, we proceed not from an object, but from a given configuration. As they say, they asked for it and got it. Below is a snippet of code:
<?php class ArrayAdapter { public function toArray($object) { $result = array(); $className = $this->getFullClassName($object); $metadata = $this->metadataFactory->getMetadataForClass($className); foreach ($metadata->getProperties() as $property) { //handle value $result[$property->getSerializedName()] = $value; } return $result; } public function toObject(array $data, $object) { $className = $this->getFullClassName($object); $metadata = $this->metadataFactory->getMetadataForClass($className); foreach ($metadata->getProperties() as $property) { //handle value and set it to object } return $object; } } ?> To handle the values ​​for serialization / deserialization, the private method handleValue (about 60 lines) is currently used.
Initially, to convert settings from a yml file to an object, I wanted to write something of my own, since the task is very simple. You need to abstract from two things: the location of the configuration files (they can be stored anywhere: the file system, memory, database) and their format (yml, json, ini, xml). At the same time, as the Thimlide noted, “we will not parse the * .yml file every time”, so we need a cache. Its interface is known to all: put, get, remove. But I stopped in time and remembered the metadata library, and again rested on Reflection that I did not need. Therefore, with minor corrections, I took this part of the code from there, saluting Johannes.
As a result, I was lost for a team of 33 hours, but the simple serializer library and the OpensoftSimpleSerializerBundle bundle were written .
Library dependencies:
Symfony / yaml component
Requirements:
- for the object being serialized, the rules of serialization should be described;
- the object to be serialized must have setters and getters for attributes.
Advantages compared to JMSSerializerBundle:
- does not have a number of urgent problems, such as: processing attributes with a null value, deserializing data into an existing object;
- setting the date formatting in the configuration;
- does not have a binding to the symfony 2 framework.
Naturally, this is not ideal. JMSSerializerBundle is more functional and has less restrictions. But, in our case, there is a clear bias to use when implementing the REST API. When we deal with REST in large applications, we will not be able to do without implementing the DTO pattern, since never will our models be explicitly reflected in the API. And, accordingly, the Assembler will most likely use the setter / getter methods when converting DTO to DomainObject. And the second requirement disappears by itself. The first, I think, is also doing everything. After all, by providing API to customers, we tell them what and in what format we accept or give. And it seems illogical if we will be missing these rules. As for the response in different formats, in our project we are dealing with json. It is very simple and convenient. XML, in my opinion, is already a bit old for use in REST. The HTML format offered by the FOSRestBundle is too contrived, I cannot imagine its use. For the reliability of the REST API meet behat tests (hello and thanks to Konstantin aka everzet ), I hope davert will not be offended, since the codeception is really cool.
It is worth talking separately about the possibility of configuring serialization. Options are minimized, although it is not difficult to add them. The file with configs is generally similar to JMSSerializerBundle:
MyBundle\Model\Order properties: id: expose: true serialized_name: id type: integer The “expose” option is “false” by default, so for all attributes that we want to serialize, it must be specified. This is my personal preference, which is not allowed - something is forbidden. Perhaps affected the enthusiasm of the ACL in his youth. In principle, you can embed a full-fledged exlude / expose, if you wish. “Serialized_name” is an optional parameter; if it is not specified, the attribute will be serialized according to its name. The last option “type” is also optional. If it is not specified, then no actions with the attribute value will be performed. Otherwise, it will be cast to the specified type. Possible types: integer, boolean, double, string, array, T, array <T>, DateTime, DateTime <format>.
The main differences from the JMSSerializerBundle are the absence of an ArrayCollection and the presence of DateTime <format>. The first is missing fundamentally, since its presence adds the dependence of the library on Doctrine / Orm. If you are using the symfony 2 framework, then at 90% it will be satisfied. And what about the other 10 percent, or those who do not use symfony? Moreover, if we are talking about applying to the DTO, then there almost certainly will not be any collections, there is no point in them. But this problem is solved in the future. The second feature is the ability to specify how to format the date. I remind you that the DateTime entry assumes that you have a DateTime object. If I'm not mistaken, in JMSSerializerBundle you can create a custom handler for the date and specify the format in it. Everything is easier here; directly in the configuration, you can specify a DateTime class constant, for example, “ISO8601”, or a string with the necessary time formatting.
You can create a TODO list, although, to be honest, I do not see any urgent need for this:
1) add the ability to store the serialization configuration in other formats (annotation, xml);
2) add the ability to serialize data in formats other than json;
3) add a variety of options to the configuration;
4) to refactor the code in handleValue, incorporating the Visitor pattern, or Chain of Responsobility, or something else, thereby adding the ability to create a custom value handler;
5) something else.
Acknowledgments: I especially want to thank my team leader for allowing me to “withdraw into myself” and for my patience during the difficult pre-release time for the project, as well as the team that destroyed the bugs while I was high :). Thanks to Johannes, Benjamin - you are really cool guys, and we don’t like to reinvent the wheel, but we had no way out.
Source: https://habr.com/ru/post/150842/
All Articles