Binary serialization in Unity3d
Faced with a rather trivial problem. Serialize and de-serialize data.
There is a client server application. Client - Unity3d server PhotonServer. There is a model that must be equivalent on both the client and the server. It is required to synchronize the state of the model and, possibly, additional classes.
The most logical solution is to use a binary protocol. This is a clear favorite - ptotobuf (using proto-net 668). It does not support web build, but it is a valid victim. Mark out the required classes. I check. Everything works, small size and fast in work. Gorgeous. But!
At one point, Protobuf spat out an ekzepchen, saying that such a class was not found. Like this?
Bug detail with sample code.
')
He began to solve this problem in various ways. There is an option to feed the Protobuf types. Which is no longer good. You can make a lot of mistakes or forget to specify one or another type. Moreover, Protobuf does not support multi-dimensional arrays.
Sadly, but Protobuf will have to the side. By the way, I once tried to use Protobuf in a bunch of php and Unity. From the php side, the implementation of Protobuf turned out to be quite buggy. As a result, in php and Unity used json. It worked, because between php and Unity there were pretty simple data structures.
There is another remarkable serializer. There are implementations for a huge number of languages. Wonderful. Decided to try. The primitive type was serializing normally. The size is 18 bytes against my 41 bytes, against 19 bytes of protobuf and against 44 bytes of json. Excellent result. What is the trick? The official site has an example of how it actually packs everything. Here is the link .
But a complex example that will not be further mastered by the message pack and protobuf.
Examples of errors.
So, Protobuf is not suitable. What to use? Json? Why not. Here is the second problem: Jason does not know how to serialize interface type fields and abstract classes. It does not matter, using Google found how to "teach" him to do it. In the final file there was data on the type of the field and its data (with the assembly indicated, this is important; why - it is written further). But for deserialization, this field is for some reason zero. Google again. After all, if taught to serialize, then you can deserialize. It turns out the same crutch as with Protobuf. This option is not suitable. I used the JSON assembly .NET For Unity, which is in the assetmarket.
Bottom line: Json is good for non-complex structures. But when there are fields like abstract class or interface, problems arise with it.
In any variation of xml - quite cumbersome. Therefore, I decided not to consider. Although part of the project in xml. For example, the localization system.
Decided to turn to standard tools. Marked up the code, serialized. Success! A large file size, however, is not good. Do not worry, go through more and compression. Used LZMA. Won a little in size, but lost on the speed of work. Valid sacrifice. Now build. Drumroll. Web not supported, trouble ...
Now arrange the exchange between the client and the server. And ... Another FAIL. The fact is that the assemblies of the classes are different, although the classes are the same. In a unit its own assembly on its own photon. Can be solved through the crutch method. Zabindit assemblies and manually rename them, but the assembly falls into a binary file. Why is she needed there?
I decided that I would come back to this method, I looked through a couple more serializers. One of them is Sharpe the serializer. I was able to serialize the interface type fields, but also prescribe the assembly and is not supported on the web. Then I decided to first formulate the requirements for the serializer.
The required serializer should be able to:
From those serializers that I tried. These requirements more or less corresponded only to Json and protobuf of the earlier version.
Began to google. But to no avail. What to do?
Using a standard solution is not a good option. Big size. Platform problems.
Then I decided, under the above written requirements, to write my own serializer. Why not. It turned out to be much more efficient than disassembling and testing one or another serializer.
Where better to start?
How to save objects and how to load them. In this regard, I liked the protobuf-net 668 approach. Namely, mark the required fields and properties. Also mark and methods that will be called before serialization and after deserialization.
First you need to save the map. Namely - the key and type. So that on this map you can then restore the object. For standard types, the value is <0 and for user types, respectively> 0. Key size is int16.
Collections rendered into separate tags to logically separate.
Now you need to get all the fields and properties. Pack them in your structure to assign tags to them.
Since there may be arrays, I add information about the array measure to the container.
I did not specifically divide into a container of type a collection and a container of type an object. All data is presented as a set. What is an array, if the object means the rank and dimension in zeros.
Now you need a container in which the object itself will be restored.
Further. With data structures complete. Now you need to fill them.
First type map and object map. For this you need a class that will unite all this. I create additional abstract classes for writing and reading. This is in case, for example, if you need to add another format. Same json or xml.
Perhaps, then they will have additional methods to write to the stream and read from the stream. But so far it is not necessary. Now we glue it all into one class.
Is done. We now turn to the tests. Caution many data sets.
Video test:
In terms of file size, of course, Protobuf and messagepack lose. After all, I save the type map and do not use clever frauds with bit shifting or string conversion “byte [] bytes = Encoding.UTF7.GetBytes ((string) data.value)”. This is an additional burden, perhaps later expand in the form of variation. Tested data exchange between photon and unit. Works as expected. After all, I create a type relative to the assembly, which is a parameter in the Deserialize method.
Ready-made solutions, which are so many on the Internet, did not fit the requirements. So I had to invent a bicycle. Who justified the time spent on him. It can be expanded to improve.
If you use primitive types, then any of the considered serializers will suit you. For primitives, I would still prefer Protobuf. But for complex data types, ready-made solutions are not always suitable.
Protobuf
Unity3d json
MSDN binary formatter
Message pack
Checkmarks
Task
There is a client server application. Client - Unity3d server PhotonServer. There is a model that must be equivalent on both the client and the server. It is required to synchronize the state of the model and, possibly, additional classes.
Decision
Protobuf
The most logical solution is to use a binary protocol. This is a clear favorite - ptotobuf (using proto-net 668). It does not support web build, but it is a valid victim. Mark out the required classes. I check. Everything works, small size and fast in work. Gorgeous. But!
At one point, Protobuf spat out an ekzepchen, saying that such a class was not found. Like this?
Bug detail with sample code.
')
He began to solve this problem in various ways. There is an option to feed the Protobuf types. Which is no longer good. You can make a lot of mistakes or forget to specify one or another type. Moreover, Protobuf does not support multi-dimensional arrays.
Sadly, but Protobuf will have to the side. By the way, I once tried to use Protobuf in a bunch of php and Unity. From the php side, the implementation of Protobuf turned out to be quite buggy. As a result, in php and Unity used json. It worked, because between php and Unity there were pretty simple data structures.
Message pack
There is another remarkable serializer. There are implementations for a huge number of languages. Wonderful. Decided to try. The primitive type was serializing normally. The size is 18 bytes against my 41 bytes, against 19 bytes of protobuf and against 44 bytes of json. Excellent result. What is the trick? The official site has an example of how it actually packs everything. Here is the link .
Example
[Serializable, ProtoContract()] public class TTT { [TDataMember, ProtoMember(1)] public string s = "compact"; [TDataMember, ProtoMember(2)] public bool f = true; [TDataMember, ProtoMember(3)] public string s2 = "schema"; [TDataMember, ProtoMember(4)] public short i = 0; } But a complex example that will not be further mastered by the message pack and protobuf.
Examples of errors.
Message pack
PlatformNotSupportedException: On-the-fly enum serializer generation is not supported in Unity iOS. Use pre-generated serializer instead. MsgPack.Serialization.ReflectionSerializers.ReflectionSerializerHelper.CreateReflectionEnuMessagePackSerializer[State] (MsgPack.Serialization.SerializationContext context) MsgPack.Serialization.MessagePackSerializer.CreateReflectionInternal[State] (MsgPack.Serialization.SerializationContext context) MsgPack.Serialization.MessagePackSerializer.CreateReflectionInternal (MsgPack.Serialization.SerializationContext context, System.Type targetType) MsgPack.Serialization.SerializationContext.GetSerializer (System.Type targetType, System.Object providerParameter) MsgPack.Serialization.ReflectionSerializers.ReflectionSerializerHelper.GetMetadata (MsgPack.Serialization.SerializationContext context, System.Type targetType, System.Func`2[]& getters, System.Action`2[]& setters, System.Reflection.MemberInfo[]& memberInfos, MsgPack.Serialization.DataMemberContract[]& contracts, MsgPack.Serialization.IMessagePackSerializer[]& serializers) MsgPack.Serialization.ReflectionSerializers.ReflectionObjectMessagePackSerializer`1[TestS+TestC]..ctor (MsgPack.Serialization.SerializationContext context) MsgPack.Serialization.MessagePackSerializer.CreateReflectionInternal[TestC] (MsgPack.Serialization.SerializationContext context) MsgPack.Serialization.SerializationContext.GetSerializer[TestC] (System.Object providerParameter) MsgPack.Serialization.SerializationContext.GetSerializer[TestC] () /// [,] string ArgumentException: 'System.String[,]' is not compatible for 'System.String[]'. Parameter name: objectTree MsgPack.Serialization.MessagePackSerializer`1[System.String[]].MsgPack.Serialization.IMessagePackSerializer.PackTo (MsgPack.Packer packer, System.Object objectTree) MsgPack.Serialization.ReflectionSerializers.ReflectionObjectMessagePackSerializer`1[TestS+TestC].PackMemberValue (MsgPack.Packer packer, .TestC objectTree, Int32 index) MsgPack.Serialization.ReflectionSerializers.ReflectionObjectMessagePackSerializer`1[TestS+TestC].PackToCore (MsgPack.Packer packer, .TestC objectTree) MsgPack.Serialization.MessagePackSerializer`1[TestS+TestC].PackTo (MsgPack.Packer packer, .TestC objectTree) MsgPack.Serialization.MessagePackSerializer`1[TestS+TestC].Pack (System.IO.Stream stream, .TestC objectTree) /// etc SerializationException: Non generic collection may contain only MessagePackObject type. MsgPack.Serialization.DefaultSerializers.NonGenericEnumerableSerializerBase`1[T].PackToCore (MsgPack.Packer packer, .T objectTree) MsgPack.Serialization.MessagePackSerializer`1[T].MsgPack.Serialization.IMessagePackSerializer.PackTo (MsgPack.Packer packer, System.Object objectTree) MsgPack.Serialization.ReflectionSerializers.ReflectionCollectionSerializer`1[System.Collections.ArrayList].PackToCore (MsgPack.Packer packer, System.Collections.ArrayList objectTree) MsgPack.Serialization.MessagePackSerializer`1[System.Collections.ArrayList].MsgPack.Serialization.IMessagePackSerializer.PackTo (MsgPack.Packer packer, System.Object objectTree) MsgPack.Serialization.ReflectionSerializers.ReflectionObjectMessagePackSerializer`1[TestS+TestC].PackMemberValue (MsgPack.Packer packer, .TestC objectTree, Int32 index) MsgPack.Serialization.ReflectionSerializers.ReflectionObjectMessagePackSerializer`1[TestS+TestC].PackToCore (MsgPack.Packer packer, .TestC objectTree) MsgPack.Serialization.MessagePackSerializer`1[TestS+TestC].PackTo (MsgPack.Packer packer, .TestC objectTree) MsgPack.Serialization.MessagePackSerializer`1[TestS+TestC].Pack (System.IO.Stream stream, .TestC objectTree) Protobuf-net
NotSupportedException: Multi-dimension arrays are supported ProtoBuf.Meta.MetaType.ResolveListTypes (ProtoBuf.Meta.TypeModel model, System.Type type, System.Type& itemType, System.Type& defaultType) ProtoBuf.Meta.MetaType.ApplyDefaultBehaviour (Boolean isEnum, ProtoBuf.ProtoMemberAttribute normalizedAttribute) ProtoBuf.Meta.MetaType.ApplyDefaultBehaviour () ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto (System.Type type, Boolean demand, Boolean addWithContractOnly, Boolean addEvenIfAutoDisabled) ProtoBuf.Meta.RuntimeTypeModel.GetKey (System.Type type, Boolean demand, Boolean getBaseKey) Json
So, Protobuf is not suitable. What to use? Json? Why not. Here is the second problem: Jason does not know how to serialize interface type fields and abstract classes. It does not matter, using Google found how to "teach" him to do it. In the final file there was data on the type of the field and its data (with the assembly indicated, this is important; why - it is written further). But for deserialization, this field is for some reason zero. Google again. After all, if taught to serialize, then you can deserialize. It turns out the same crutch as with Protobuf. This option is not suitable. I used the JSON assembly .NET For Unity, which is in the assetmarket.
Bottom line: Json is good for non-complex structures. But when there are fields like abstract class or interface, problems arise with it.
XML
In any variation of xml - quite cumbersome. Therefore, I decided not to consider. Although part of the project in xml. For example, the localization system.
BinaryFormatter
Decided to turn to standard tools. Marked up the code, serialized. Success! A large file size, however, is not good. Do not worry, go through more and compression. Used LZMA. Won a little in size, but lost on the speed of work. Valid sacrifice. Now build. Drumroll. Web not supported, trouble ...
Now arrange the exchange between the client and the server. And ... Another FAIL. The fact is that the assemblies of the classes are different, although the classes are the same. In a unit its own assembly on its own photon. Can be solved through the crutch method. Zabindit assemblies and manually rename them, but the assembly falls into a binary file. Why is she needed there?
I decided that I would come back to this method, I looked through a couple more serializers. One of them is Sharpe the serializer. I was able to serialize the interface type fields, but also prescribe the assembly and is not supported on the web. Then I decided to first formulate the requirements for the serializer.
Primitive type test
TTT c = new TTT(); TSerizalization serizalization = new TSerizalization(); bytes = serizalization.Serizalize(c, true); System.IO.File.WriteAllBytes("d:\\s.dat", bytes); Debug.LogError("T complete " + bytes.Length ); json = JsonConvert.SerializeObject(c); System.IO.File.WriteAllText("d:\\s.json", json); Debug.LogError("J complete " + json.Length); System.Runtime.Serialization.Formatters.Binary.BinaryFormatter formater = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter(); formater.AssemblyFormat = System.Runtime.Serialization.Formatters.FormatterAssemblyStyle.Full; System.IO.MemoryStream mstream = new System.IO.MemoryStream(); formater.Serialize(mstream, c); Debug.LogError("B complete " + mstream.ToArray().Length); System.IO.File.WriteAllBytes("d:\\s2.dat", mstream.ToArray()); mstream = new System.IO.MemoryStream(); var serializer = MsgPack.Serialization.SerializationContext.Default.GetSerializer<TTT>(); serializer.Pack(mstream, c); System.IO.File.WriteAllBytes("d:\\s3.dat", mstream.ToArray()); Debug.LogError("M complete " + mstream.ToArray().Length); mstream = new System.IO.MemoryStream(); ProtoBuf.Serializer.Serialize<TTT>(mstream, c); System.IO.File.WriteAllBytes("d:\\s4.dat", mstream.ToArray()); Debug.LogError("P complete " + mstream.ToArray().Length); Serializer requirements
The required serializer should be able to:
- Serialize to binary format;
- Serialize to small size;
- Serialize classes using specific assemblies (the assembly information is not necessarily needed in the file);
- Serialize custom classes;
- Maintain arrays;
- Work in the web version of the unit;
- Work with a photon.
From those serializers that I tried. These requirements more or less corresponded only to Json and protobuf of the earlier version.
Own serializer
Began to google. But to no avail. What to do?
Using a standard solution is not a good option. Big size. Platform problems.
Then I decided, under the above written requirements, to write my own serializer. Why not. It turned out to be much more efficient than disassembling and testing one or another serializer.
Where better to start?
How to save objects and how to load them. In this regard, I liked the protobuf-net 668 approach. Namely, mark the required fields and properties. Also mark and methods that will be called before serialization and after deserialization.
Map
First you need to save the map. Namely - the key and type. So that on this map you can then restore the object. For standard types, the value is <0 and for user types, respectively> 0. Key size is int16.
Map
public class TMap { public Dictionary<Type, short> StandartTypes { get; protected set; } public Dictionary<Type, short> DataBase { get; protected set; } public Dictionary<short, Type> DataBaseTags { get; protected set; } ... } Collections rendered into separate tags to logically separate.
TData many objects
Now you need to get all the fields and properties. Pack them in your structure to assign tags to them.
Many objects
public class TData : TContainerBase { public object value; public List<TData> childrens = new List<TData>(); ... } Since there may be arrays, I add information about the array measure to the container.
Base Container Description
public class TContainerBase { public short Tag { get; protected set; } public int ArrayRank { get; protected set; } public List<int> ArrayDimension { get; protected set; } ... I did not specifically divide into a container of type a collection and a container of type an object. All data is presented as a set. What is an array, if the object means the rank and dimension in zeros.
Now you need a container in which the object itself will be restored.
Container
public class TContainer : TContainerBase { public int Size { get; protected set; } public List<object> List { get; protected set; } ... Further. With data structures complete. Now you need to fill them.
First type map and object map. For this you need a class that will unite all this. I create additional abstract classes for writing and reading. This is in case, for example, if you need to add another format. Same json or xml.
Reading write
public abstract class TReaderBase { public abstract T Read<T>(byte[] bytes, Assembly assembly); } public abstract class TWriterBase { public abstract byte[] Write(TMap map, TData data); } Perhaps, then they will have additional methods to write to the stream and read from the stream. But so far it is not necessary. Now we glue it all into one class.
T serialization
public class TSerizalization { protected TMap map; protected TWriterBase writer; protected TReaderBase reader; public TSerizalization() { writer = new TBinaryWriter(); reader = new TBinaryReader(); } public virtual byte[] Serialize(object target, bool callBeforeSerializationMethods = false); public virtual T Deserialize<T>(byte[] bytes, Assembly assembly, bool callAfterDeserializationMethods = false); protected virtual TData Read(object obj) } Test
Is done. We now turn to the tests. Caution many data sets.
Test
public interface IClass { } [System.Serializable ] public class TestC : IClass { [To2dnd.TDataMember] public int a = 10; [To2dnd.TDataMember] public int b = 12; [To2dnd.TDataMember] public string s= "Hello World"; [To2dnd.TDataMember] public State state = State.Close; [To2dnd.TDataMember] public DateTime dt = new DateTime(); [To2dnd.TDataMember] public Type type = typeof(IClass); [To2dnd.TDataMember] public string[,] arr = new string[,] { {"1111", "2222", "3333", "4444" }, {"aaaa", "bbbb", "cccc", "dddd" }, {"321", "32", "2qfs", "12f" } }; [To2dnd.TDataMember] public object classD = new TestC2(); [To2dnd.TDataMember] public TestC1[] array1 = new TestC1[] { new TestC1(), new TestC2(), new TestC2() }; [To2dnd.TDataMember] public ArrayList arr2 = new ArrayList( new string[]{ "list1", "list2" }); [To2dnd.TDataMember] public List<string> list = new List<string>() { "list Item 1", "List Item 2" }; [To2dnd.TDataMember] public Dictionary<string, int> dic = new Dictionary<string, int>() { {"one", 1}, {"two", 2}, {"three", 3}, {"four", 4} }; [To2dnd.TDataMember] public Hashtable ht = new Hashtable() { {"H one", 1}, {"H two", 2}, {"H three", 3}, {"H four", 4} }; [To2dnd.TDataMember] public SortedList<string, int> sl = new SortedList<string, int>() { {"S one", 1}, {"S two", 2}, {"S three", 3}, {"S four", 4} }; [To2dnd.TDataMember] public Dictionary<string, List<string>> dic3 = new Dictionary<string,List<string>>() { {">> 1", new List<string>(){"a1", "a2", "a3"} }, {">> 2", new List<string>(){"b1", "b2", "b3"} }, {">> 3", new List<string>(){"c1", "c2", "c3"} } }; [To2dnd.TDataMember] public List<List<string>> l = new List<List<string>>() { new List<string>(){"a1", "a2", "a3"}, new List<string>(){"b1", "b2", "b3"}, }; [ProtoMember(16)] public Dictionary<string, Dictionary<string, string>> dic4 = new Dictionary<string, Dictionary<string, string>>() { {">> 1", new Dictionary<string, string>() { { "a1", "a2"}, { "a2", "a3"} } }, {">> 2", new Dictionary<string, string>() { { "a1", "a2"}, { "a2", "a3"} } } }; [ProtoMember(17)] public Dictionary<string, Dictionary<string, string>> Dic4 {get; protected set;} [ProtoMember(18)] public Dictionary<string, object> dic333 = new Dictionary<string, object>() { {":@", new List<string>(){"1", "2", "3"}}, {":@2", new TestC2()}, {":@222", "sff"} }; public TestC() { Dic4 = new Dictionary<string,Dictionary<string,string>>() { {">> 1", new Dictionary<string, string>() { { "a1", "a2"}, { "a2", "a3"} } }, {">> 2", new Dictionary<string, string>() { { "a1", "a2"}, { "a2", "a3"} } } }; } } [Serializable] public class TestC1 : IClass { [To2dnd.TDataMember] public float value1 = 10; [To2dnd.TDataMember] public float value2 = 12; } [Serializable ] public class TestC2 : TestC1 { [To2dnd.TDataMember] public float a1 = 10; [To2dnd.TDataMember] public float b2 = 12; [To2dnd.TDataMember] public string str = "Class 1"; [To2dnd.TDataMember] public State state = State.Close; public TestC2() { } [TAfterDeserialization] public void After() { } [TBeforeSerialization] public void Before() { } } public class TestC33 { [To2dnd.TDataMember] public float b2 = 12; [To2dnd.TDataMember] public TestC2 tt = new TestC2(); [ To2dnd.TDataMember] public TestC1[] array1 = new TestC1[] { new TestC1(), new TestC2(), new TestC2() }; [To2dnd.TDataMember] public object classD = new TestC2(); [To2dnd.TDataMember] public Type type = typeof(IClass); } Video test:
Total
In terms of file size, of course, Protobuf and messagepack lose. After all, I save the type map and do not use clever frauds with bit shifting or string conversion “byte [] bytes = Encoding.UTF7.GetBytes ((string) data.value)”. This is an additional burden, perhaps later expand in the form of variation. Tested data exchange between photon and unit. Works as expected. After all, I create a type relative to the assembly, which is a parameter in the Deserialize method.
Ready-made solutions, which are so many on the Internet, did not fit the requirements. So I had to invent a bicycle. Who justified the time spent on him. It can be expanded to improve.
Conclusion
If you use primitive types, then any of the considered serializers will suit you. For primitives, I would still prefer Protobuf. But for complex data types, ready-made solutions are not always suitable.
Links
Protobuf
Unity3d json
MSDN binary formatter
Message pack
Checkmarks
Source: https://habr.com/ru/post/255399/
All Articles