Message dispatching on D
Many game developers are faced with the problem of describing and implementing the client and server communication protocol, especially if they write their bicycles for working with sockets. Below I will talk about my attempt to solve the problem as elegantly and conveniently as possible for further use and scaling of the application. There will be a lot of compile-time'a with automatic code generation, gently seasoned with a pinch of run-time'a.
The client and the server are constantly exchanged messages, but these messages must first be prepared, forwarded and then restored to a readable form. Brief diagram below:
The main problem arises between the stage of reading the message and deserialization, a stream of bytes comes to the receiver, and for correct deserialization you need to know the structure of the message, that is, the type. All operations on types ended in compile-time and we no longer have the help of the compiler. The very first, most brutal solution that comes to mind is to write a huge switch linking the message id and a specific function to unpack the message. I think it’s not necessary to explain why this decision leads to a headache when processing the protocol and a huge number of difficult-to-find errors. This problem will be solved.
')
First you need to determine what we want to get:
Our project uses its own serializer, which also actively uses compile-time (This is a topic for a separate post). Let us agree that we have a certain black box that can translate classes and their fields into bytes and back again with such calls:
Also, for simplicity, we naively assume that the messages are not encrypted and the serializer solves all security problems if the message does not match the stated structure, throws an exception, and we ignore the problematic message.
All the code that goes further was tested on dmd 2.060 and probably no longer compiled to 2.059 (a very unpleasant children's disease D2).
Each message is a certain class that has a functional operator overloaded and there is a constructor without parameters (a requirement for deserialization). The first requirement is easy to formalize; any message must implement this interface:
The second constructor is needed to build the message, about this and about checking the presence of the constructor without parameters below.
In C ++, I would use many, many structures with template-specific parameters, but in D there are other ways to execute code in compile-time. I will use templates and mixins so that as little as possible the compile-time code is settled in the executable file. In total, all the code will be in the template mixin, it can be easily used again in another application or in another version of the same.
IndexType is the type of index we will use. SerializerBackend is the backend for the serializer, it is possible that another application will use a different serialization mechanism in bytes or even not in bytes, but xml / json.
pairs ... - The most interesting parameter, pairs will be recorded here: id and message type. Example below:
But the user can stuff everything into pairs, break this fragile agreement, and then the problems will not keep themselves waiting. It is necessary to check the correctness. Therefore, we will insert into the template one more template that will run through the pairs and stop the compilation with a beautiful and understandable error message.
There may be incomprehensible calls to __traits (compiles, sometext) , this is an explicit request to the compiler to check whether sometext is compiled at all or not. You can read more about embedded Traits here . And right after the template declaration, we call it through static assert . It would be possible to simply call this template, but the compiler swears at obviously meaningless expressions, which sometimes hinders a little.
An attentive reader (if anyone got to this line at all) probably noticed that I did not define the CountValInList pattern, which counts the number of occurrences of the value in the list.
Great, all improper uses are clipped and processed correctly. According to such error messages, it is quite possible to find the right way to use the scientific method of typing (this will not save writing the documentation!). Now you need to think about the task itself. We need a compromise between usability and speed, stop, we can get this and that at the same time! We will generate a giant switch automatically without the participation of the programmer:
This template will generate a string similar to this:
It now remains to mix the resulting string into the function for dispatching:
What the call to this function will look like in code:
Actually the most difficult part is written, there were only any goodies for convenient design of the message. Nobody wants to do it manually ?! It is much more convenient to do it like this:
No id, no other extra stuff. The parameters are immediately passed to the message constructor, and the message is serialized into a stream of bytes. It remains to write this ... You need to be able to search for the message id by type, for this you need another template:
At this point, my tiny number of publics should have the thought that I suffer from a mania for functional programming. I respect all the paradigms, but in the compile-time templates there is no mutable state, so a functional style naturally arises here. Now it is not difficult to construct a message, knowing only its type:
Now that we have this clever system, we need to test it in practice. For this, I wrote unittest:
For the integrity of the picture below is the full source code under the Boost license. For normal operation, the module needs a serializer, you can screw your own or use Orange.
Formulation of the problem
The client and the server are constantly exchanged messages, but these messages must first be prepared, forwarded and then restored to a readable form. Brief diagram below:
The main problem arises between the stage of reading the message and deserialization, a stream of bytes comes to the receiver, and for correct deserialization you need to know the structure of the message, that is, the type. All operations on types ended in compile-time and we no longer have the help of the compiler. The very first, most brutal solution that comes to mind is to write a huge switch linking the message id and a specific function to unpack the message. I think it’s not necessary to explain why this decision leads to a headache when processing the protocol and a huge number of difficult-to-find errors. This problem will be solved.
')
First you need to determine what we want to get:
- Link a message id to a specific message handler class once. And that's all, never remember further about id. Something like this:
0 AMessage
1 BMessage
2 CMessage - Correctly handle usage errors and stop attempts to spoil the code at the compilation stage. For example, in C ++ it is almost impossible to get clear error messages when dealing with compile-time structures.
- Simple use, one function call and a stream of bytes turned into a message ready for processing.
Dependencies
Our project uses its own serializer, which also actively uses compile-time (This is a topic for a separate post). Let us agree that we have a certain black box that can translate classes and their fields into bytes and back again with such calls:
auto stream = serialize!(ByteBackend)(SomeObject, "name"); auto object = deserialize!(ByteBackend, SomeClass)(stream, "name"); Also, for simplicity, we naively assume that the messages are not encrypted and the serializer solves all security problems if the message does not match the stated structure, throws an exception, and we ignore the problematic message.
All the code that goes further was tested on dmd 2.060 and probably no longer compiled to 2.059 (a very unpleasant children's disease D2).
Messages
Each message is a certain class that has a functional operator overloaded and there is a constructor without parameters (a requirement for deserialization). The first requirement is easy to formalize; any message must implement this interface:
Code
Sample message:
interface Message { void opCall(); } Sample message:
class AMsg : Message { int a; string b; this() {} this(int pa, string pb) { a = pa; b = pb; } void opCall() { writeln("AMsg call with ", a, " ", b); } } The second constructor is needed to build the message, about this and about checking the presence of the constructor without parameters below.
We start to make magic
In C ++, I would use many, many structures with template-specific parameters, but in D there are other ways to execute code in compile-time. I will use templates and mixins so that as little as possible the compile-time code is settled in the executable file. In total, all the code will be in the template mixin, it can be easily used again in another application or in another version of the same.
mixin template ProtocolPool(IndexType, SerializerBackend, pairs...) { } IndexType is the type of index we will use. SerializerBackend is the backend for the serializer, it is possible that another application will use a different serialization mechanism in bytes or even not in bytes, but xml / json.
pairs ... - The most interesting parameter, pairs will be recorded here: id and message type. Example below:
mixin ProtocolPool!(int, BinaryBackend, 0, AMsg, 1, BMsg, 2, CMsg ); Error processing
But the user can stuff everything into pairs, break this fragile agreement, and then the problems will not keep themselves waiting. It is necessary to check the correctness. Therefore, we will insert into the template one more template that will run through the pairs and stop the compilation with a beautiful and understandable error message.
Code
template CheckPairs(tpairs...) { static if(tpairs.length > 1) { static assert(__traits(compiles, typeof(tpairs) ), "ProtocolPool expected index first, but got some type"); static assert(is(typeof(tpairs[0]) == IndexType), "ProtocolPool expected index first of type "~ IndexType.stringof~ " not a "~typeof(tpairs[0]).stringof); static assert(is(tpairs[1] : Message), "ProtocolPool expected class implementing Message"~ " interface following index not a "~tpairs[1].stringof); static assert(CountValInList!(tpairs[0], pairs) == 1, "ProtocolPool indexes must be unique! One message,"~ "one index."); enum CheckPairs = CheckPairs!(tpairs[2..$]); } else { static assert(tpairs.length == 0, "ProtocolPool expected even number of parameters. Index and message type."); enum CheckPairs = 0; } } There may be incomprehensible calls to __traits (compiles, sometext) , this is an explicit request to the compiler to check whether sometext is compiled at all or not. You can read more about embedded Traits here . And right after the template declaration, we call it through static assert . It would be possible to simply call this template, but the compiler swears at obviously meaningless expressions, which sometimes hinders a little.
Code
mixin template ProtocolPool(IndexType, SerializerBackend, pairs...) { template CheckPairs(tpairs...) { // , } static assert(CheckPairs!pairs == 0, "Parameters check failed! If code works well, you never will see this message!"); } An attentive reader (if anyone got to this line at all) probably noticed that I did not define the CountValInList pattern, which counts the number of occurrences of the value in the list.
Code
// returns count of val occurenes in list template CountValInList(IndexType val, list...) { static if(list.length > 1) { static if(list[0] == val) enum CountValInList = 1 + CountValInList!(val, list[2..$]); else enum CountValInList = CountValInList!(val, list[2..$]); } else enum CountValInList = 0; } Code Generation
Great, all improper uses are clipped and processed correctly. According to such error messages, it is quite possible to find the right way to use the scientific method of typing (this will not save writing the documentation!). Now you need to think about the task itself. We need a compromise between usability and speed, stop, we can get this and that at the same time! We will generate a giant switch automatically without the participation of the programmer:
Code
// generating switch template GenerateSwitch() { template GenerateSwitchBody(tpairs...) { static if(tpairs.length > 0) { enum GenerateSwitchBody = "case("~to!string(tpairs[0])~ "): return cast(Message)(func!(SerializerBackend, "~ tpairs[1].stringof~")(args)); break; \n" ~ GenerateSwitchBody!(tpairs[2..$]); } else enum GenerateSwitchBody = ""; } enum GenerateSwitch = "switch(id)\n{\n"~ GenerateSwitchBody!(pairs) ~ "default: " ~ " break;\n}"; } This template will generate a string similar to this:
Code
switch(id) { case(0): return cast(Message)(func!(SerializerBackend, AMsg)(args)); break; case(1): return cast(Message)(func!(SerializerBackend, BMsg)(args)); break; case(2): return cast(Message)(func!(SerializerBackend, CMsg)(args)); break; default: break; } It now remains to mix the resulting string into the function for dispatching:
Code
// , nested class , private class dummyClass {} // func - , args Message dispatchMessage(alias func, T...)(IndexType id, T args) { static assert(__traits(compiles, func!(SerializerBackend, dummyClass)(args)), "ChooseMessage func must be callable with got args " ~T.stringof); // , //pragma(msg, GenerateSwitch!()); mixin(GenerateSwitch!()); throw new Exception( "Cannot find corresponding message for id "~to!string(id)~"!"); } What the call to this function will look like in code:
Code
void readMsg(Stream stream) { int id; stream.read(id); writeln("Got message id is ",id); auto message = dispatchMessage!(deserialize)(id, stream, "MSG"); writeln("Calling message"); message(); } Actually the most difficult part is written, there were only any goodies for convenient design of the message. Nobody wants to do it manually ?! It is much more convenient to do it like this:
auto stream = constructMessage!AMsg(10, "Hello World!"); No id, no other extra stuff. The parameters are immediately passed to the message constructor, and the message is serialized into a stream of bytes. It remains to write this ... You need to be able to search for the message id by type, for this you need another template:
Code
template FindMessageId(Msg, tpairs...) { static if(tpairs.length > 0) { static if(is(tpairs[1] == Msg)) enum FindMessageId = tpairs[0]; else enum FindMessageId = FindMessageId!(Msg, tpairs[2..$]); } else static assert(false, "Cannot find id for message "~ Msg.stringof~". Check protocol list."); } At this point, my tiny number of publics should have the thought that I suffer from a mania for functional programming. I respect all the paradigms, but in the compile-time templates there is no mutable state, so a functional style naturally arises here. Now it is not difficult to construct a message, knowing only its type:
Code
Stream constructMessage(Msg, T...)(T args) { static assert(is(Msg : Message), Msg.stringof~ " must implement Message interface!"); static assert(__traits(compiles, new Msg(args)), Msg.stringof~ " should implement constructor with formal parameters "~ T.stringof); auto msg = new Msg(args); IndexType sendId = FindMessageId!(Msg, pairs); auto stream = serialize!SerializerBackend(msg, "MSG"); auto fullStream = new MemoryStream; fullStream.write(sendId); fullStream.copyFrom(stream); fullStream.position = 0; return fullStream; } Using
Now that we have this clever system, we need to test it in practice. For this, I wrote unittest:
Code
version(unittest) { class AMsg : Message { int a; string b; this() {} this(int pa, string pb) { a = pa; b = pb; } void opCall() { writeln("AMsg call with ", a, " ", b); } } class BMsg : Message { double a; double b; this() {} this(double pa, double pb) { a = pa; b = pb; } void opCall() { writeln("BMsg call with ", a, " ", b); } } class CMsg : Message { double a; string s; this() {} this(double pa, string ps) { a = pa; s = ps; } void opCall() { writeln("CMsg call ", a, " ", s); } } mixin ProtocolPool!(int, GendocArchive, 0, AMsg, 1, BMsg, 2, CMsg ); } unittest { void readMsg(Stream stream) { int id; stream.read(id); writeln("Got message id is ",id); auto message = dispatchMessage!(deserialize)(id, stream, "MSG"); writeln("Calling message"); message(); } // serializing auto stream = constructMessage!BMsg(4.0,8.0); // sending... // got at other side readMsg(stream); stream = constructMessage!AMsg(10, "Hello World!"); readMsg(stream); stream = constructMessage!CMsg(5., "Some usefull string"); readMsg(stream); } Full source code
For the integrity of the picture below is the full source code under the Boost license. For normal operation, the module needs a serializer, you can screw your own or use Orange.
Code
// Copyright Gushcha Anton 2012. // Distributed under the Boost Software License, Version 1.0. // (See accompanying file LICENSE_1_0.txt or copy at // http://www.boost.org/LICENSE_1_0.txt) module protocol; import std.stdio; import std.conv; import std.stream; // , import util.serialization.serializer; interface Message { void opCall(); } mixin template ProtocolPool(IndexType, SerializerBackend, pairs...) { // returns count of val occurenes in list template CountValInList(IndexType val, list...) { static if(list.length > 1) { static if(list[0] == val) enum CountValInList = 1 + CountValInList!(val, list[2..$]); else enum CountValInList = CountValInList!(val, list[2..$]); } else enum CountValInList = 0; } // check pairs to be correct template CheckPairs(tpairs...) { static if(tpairs.length > 1) { static assert(__traits(compiles, typeof(tpairs) ), "ProtocolPool expected index first, but got some type"); static assert(is(typeof(tpairs[0]) == IndexType), "ProtocolPool expected index first of type "~IndexType.stringof~" not a "~typeof(tpairs[0]).stringof); static assert(is(tpairs[1] : Message), "ProtocolPool expected class implementing Message interface following index not a "~tpairs[1].stringof); static assert(CountValInList!(tpairs[0], pairs) == 1, "ProtocolPool indexes must be unique! One message, one index."); enum CheckPairs = CheckPairs!(tpairs[2..$]); } else { static assert(tpairs.length == 0, "ProtocolPool expected even number of parameters. Index and message type."); enum CheckPairs = 0; } } // generating switch template GenerateSwitch() { template GenerateSwitchBody(tpairs...) { static if(tpairs.length > 0) { enum GenerateSwitchBody = "case("~to!string(tpairs[0])~"): return cast(Message)(func!(SerializerBackend, "~tpairs[1].stringof~")(args)); break; \n" ~ GenerateSwitchBody!(tpairs[2..$]); } else enum GenerateSwitchBody = ""; } enum GenerateSwitch = "switch(id)\n{\n"~GenerateSwitchBody!(pairs) ~ `default: ` ~ " break;\n}"; } template FindMessageId(Msg, tpairs...) { static if(tpairs.length > 0) { static if(is(tpairs[1] == Msg)) enum FindMessageId = tpairs[0]; else enum FindMessageId = FindMessageId!(Msg, tpairs[2..$]); } else static assert(false, "Cannot find id for message "~Msg.stringof~". Check protocol list."); } // actual check static assert(CheckPairs!pairs == 0, "Parameters check failed! If code works well, you never will see this message!"); private class dummyClass {} Message dispatchMessage(alias func, T...)(IndexType id, T args) { static assert(__traits(compiles, func!(SerializerBackend, dummyClass)(args)), "ChooseMessage func must be callable with got args "~T.stringof); //pragma(msg, GenerateSwitch!()); mixin(GenerateSwitch!()); throw new Exception("Cannot find corresponding message for id "~to!string(id)~"!"); } Stream constructMessage(Msg, T...)(T args) { static assert(is(Msg : Message), Msg.stringof~" must implement Message interface!"); static assert(__traits(compiles, new Msg(args)), Msg.stringof~" should implement constructor with formal parameters "~T.stringof); auto msg = new Msg(args); IndexType sendId = FindMessageId!(Msg, pairs); auto stream = serialize!SerializerBackend(msg, "MSG"); auto fullStream = new MemoryStream; fullStream.write(sendId); fullStream.copyFrom(stream); fullStream.position = 0; return fullStream; } } version(unittest) { class AMsg : Message { int a; string b; this() {} this(int pa, string pb) { a = pa; b = pb; } void opCall() { writeln("AMsg call with ", a, " ", b); } } class BMsg : Message { double a; double b; this() {} this(double pa, double pb) { a = pa; b = pb; } void opCall() { writeln("BMsg call with ", a, " ", b); } } class CMsg : Message { double a; string s; this() {} this(double pa, string ps) { a = pa; s = ps; } void opCall() { writeln("CMsg call ", a, " ", s); } } mixin ProtocolPool!(int, BinaryBackend, 0, AMsg, 1, BMsg, 2, CMsg ); } unittest { void readMsg(Stream stream) { int id; stream.read(id); writeln("Got message id is ",id); auto message = dispatchMessage!(deserialize)(id, stream, "MSG"); writeln("Calling message"); message(); } // serializing auto stream = constructMessage!BMsg(4.0,8.0); // sending... // Got at other side readMsg(stream); stream = constructMessage!AMsg(10, "Hello World!"); readMsg(stream); stream = constructMessage!CMsg(5., "Some usefull string"); readMsg(stream); } Source: https://habr.com/ru/post/149809/
All Articles