Using Protocol Buffers on the .Net platform (Part 1)

I bring to your attention an introduction to using Protocol Buffers on the .Net platform in a discussion format. I will tell and show what it is and why it is needed .Net developer. The topic requires the reader to have an initial knowledge of the C # language and the SVN version control system. Since the volume of the material exceeds the average volume of topics on the habre, which do not drive habouriers into melancholy and do not force them to scroll to comments, it was decided to divide it into two parts. In the first part we will get acquainted with the basics and even write (not) a lot of code!

Hello, what is Protocol Buffers?

According to the definition on the official page, Protocol Buffers (protobuf) is a way to encode structured data in an efficient and extensible format, used by Google in almost all of its products. For most platforms, including .Net, this process is called serialization .

I use Google services, but how will protobuf help me in developing .Net applications?

Yes you are right, Google is not engaged in writing specialized libraries for .Net developers. However, there is a protobuf-net project (one of several protobuf implementations for the platform) that allows using protobuf in .Net. He is led by Marc Gravell - a regular at stackoverflow.com and a member of many other great projects . So you can always ask him a question, and he will be happy to answer it (which the author of the topic abused).

')

Why should I use this library instead of built-in tools?

When it comes to serialization in .Net, everyone usually remembers the existence of a binary formatter and xml serializer . The next thing the developers note is that the first one is fast and has a high compression ratio, but it works only within the .Net platform; and the second presents data in a human-readable format and serves as the basis for SOAP , which in turn provides cross-platform. In fact, the statement that you always have to choose between speed and portability is taken as an axiom! But protobuf can solve both problems at once.

So you claim that protobuf is not inferior to binary serialization and is also portable?

Yes it is. Let's look at a small example, and at the same time we will learn how to use protobuf-net. Suppose we have the following entities:

using System;

using ProtoBuf;



namespace Proto.Sample

{

public enum TaskPriority

{

Low,

Medium,

High

}



[ Serializable ] // <-- BinaryFormatter

[ProtoContract]

public class Task

{

[ProtoMember(1)]

public int Id { get ; set ; }



[ProtoMember(2)]

public DateTime CreatedAt { get ; set ; }



[ProtoMember(3)]

public string CreatedBy { get ; set ; }



[ProtoMember(4)]

public TaskPriority Priority { get ; set ; }



[ProtoMember(5)]

public string Content { get ; set ; }

}

}



* This source code was highlighted with Source Code Highlighter .

Protobuf-net requires the use of special attributes, which directly follows from the main feature of the format - depending on the order of the fields. Let's write a test of performance and compression ratio:

using System;

using System.Collections. Generic ;

using System.Diagnostics;

using System.IO;

using System.Runtime.Serialization.Formatters.Binary;

using ProtoBuf;



namespace Proto.Sample

{

internal class Program

{

private static void Main( string [] args)

{

var tasks = new List <Task>

{

new Task

{

Id = 1,

CreatedBy = "Steve Jobs" ,

CreatedAt = DateTime .Now,

Priority = TaskPriority.High,

Content = "Invent new iPhone"

},

new Task

{

Id = 2,

CreatedBy = "Steve Ballmer" ,

CreatedAt = DateTime .Now.AddDays(-7),

Priority = TaskPriority.Low,

Content = "Install own Skype"

}

};



Console .WriteLine( "The test of binary formatter:" );



const string file1 = "tasks1.bin" ;



TestBinaryFormatter(tasks, file1, 1000);

TestBinaryFormatter(tasks, file1, 2000);

TestBinaryFormatter(tasks, file1, 3000);

TestBinaryFormatter(tasks, file1, 4000);

TestBinaryFormatter(tasks, file1, 5000);



Console .WriteLine( "\nThe test of protobuf-net:" );



const string file2 = "tasks2.bin" ;



TestProtoBuf(tasks, file2, 1000);

TestProtoBuf(tasks, file2, 2000);

TestProtoBuf(tasks, file2, 3000);

TestProtoBuf(tasks, file2, 4000);

TestProtoBuf(tasks, file2, 5000);



Console .WriteLine( "\nThe comparision of file size:" );



Console .WriteLine( "The size of {0} is {1} bytes" , file1, ( new FileInfo(file1)).Length);

Console .WriteLine( "The size of {0} is {1} bytes" , file2, ( new FileInfo(file2)).Length);



Console .ReadKey();

}



private static void TestBinaryFormatter(IList<Task> tasks, string fileName, int iterationCount)

{

var stopwatch = new Stopwatch();

var formatter = new BinaryFormatter();

using ( var file = File .Create(fileName))

{

stopwatch.Restart();



for ( var i = 0; i < iterationCount; i++)

{

file.Position = 0;

formatter.Serialize(file, tasks);

file.Position = 0;

var restoredTasks = ( List <Task>)formatter.Deserialize(file);

}



stopwatch.Stop();



Console .WriteLine( "{0} iterations in {1} ms" , iterationCount, stopwatch.ElapsedMilliseconds);

}

}



private static void TestProtoBuf(IList<Task> tasks, string fileName, int iterationCount)

{

var stopwatch = new Stopwatch();

using ( var file = File .Create(fileName))

{

stopwatch.Restart();



for ( var i = 0; i < iterationCount; i++)

{

file.Position = 0;

Serializer.Serialize(file, tasks);

file.Position = 0;

var restoredTasks = Serializer.Deserialize< List <Task>>(file);

}



stopwatch.Stop();



Console .WriteLine( "{0} iterations in {1} ms" , iterationCount, stopwatch.ElapsedMilliseconds);

}

}

}

}



* This source code was highlighted with Source Code Highlighter .

Results:

The test of binary formatter:

1000 iterations in 423 ms

2000 iterations in 381 ms

3000 iterations in 532 ms

4000 iterations in 660 ms

5000 iterations in 814 ms



The test of protobuf-net:

1000 iterations in 1056 ms

2000 iterations in 76 ms

3000 iterations in 129 ms

4000 iterations in 152 ms

5000 iterations in 202 ms



The comparision of file size:

The size of tasks1.bin is 710 bytes

The size of tasks2.bin is 101 bytes



* This source code was highlighted with Source Code Highlighter .

As you can see, we have surpassed binary serialization not only in speed, but also in compression. The only drawback is that protobuf-net took more time to “cold start”. But you can solve this problem using the following auxiliary code:

var model = TypeModel.Create();

model.Add( typeof (Task), true );

var compiledModel = model.Compile(path);

compiledModel.Serialize(file, tasks);



* This source code was highlighted with Source Code Highlighter .

The rest of the tests and results can be found here .

OK. Regarding speed and compression, you convinced me, but how is the problem of portability solved?

You see, if there is an implementation for the platform you need, the issue of portability is removed in most cases. And protobuf implementations exist for more than 20 languages. The full list can be seen here . I note only that for some languages ​​there is more than one implementation. So you always have a choice.



UPD: Part 2