Network Protocols Design
I searched the articles on protocol design for habr and, to my surprise, found nothing. Perhaps it is worth then to share your thoughts on sabzh. I must say at once that the division into types is purely mine and may not coincide with what you find in reference books. We also agree in advance that C / C ++ is used.
Consider the main types of protocols:
These protocols are based on communication in small portions of data. The communication protocol is usually strongly typed. As an example, you can bring all known AT-commands for modems.
This type of protocol is the easiest to process (an elementary parsing of a string with isolation of data is required). But to communicate with such a protocol on more or less serious tasks is not very easy. Such protocols are well suited for sending small portions of scalar data types (strings, numbers).
Nevertheless, it is also necessary to design such protocols.
We need to transfer 10 numbers from the client to the server, and receive in response a string or a number (varies from the input data). For this we need: the “handshake” stage, the data transfer stage, the answer receiving stage.
“Manpower” : it is quite enough to send the word “HELLO” back and forth (if we know that there is a possibility to get to another client instead of the server, then we can separate the client and server greetings, for example, “HELLOCL” (from the client) and “HELLOSRV” (from server)). The processing is elementary and comes down to string comparison.
Data transfer : we take the command “SEND x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” as the sending command from the client and “OK SEND” as the server response. The processing is again elementary and comes down to string comparison and calling sscanf ().
Receiving the answer : let's agree that the server sends an “ANSWER STR string ” (if the answer is a string) or an ANSWER NUM number (if the answer is a number). The client responds with the “OK ANSWER” command.
It would seem that everything is simple and clear. BUT. After all, this way we will not be able to understand to which set of numbers the answer sent belongs. To solve this problem is simple. When sending data, we will use the commands: “SEND id NUMS x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” and “OK SEND id ”, where id is the unique identifier for this set of numbers. When responding, the commands will be as follows: “ANSWER id STR string ”, “ANSWER id NUM number ”, “OK ANSWER id ”. Such a protocol will already be exhaustive in most cases.
')
This type of protocol is the most common. Its basis is hard typing of a portion of the sent data. That is, we predetermine in advance that in all packets at such a displacement and such a length such and such data will lie (the displacement and length of some fields can also be specified in the structure, but mainly the initially specified displacements are used). These protocols are almost all low-level protocols.
The undoubted advantage of such protocols (especially under the condition of rigidly specified displacements and lengths of all fields) is the extreme simplicity of processing. We just need to check the size and execute the memcpy () command into an instance of the structure corresponding to our package.
When designing such protocols, it is necessary to remember about the features of storing structures in C. I mean what is called packing. The fact is that any instance of any structure must be aligned in memory with a certain multiplicity (the multiplicity is set for the whole program alone). By default, the multiplicity is 4 (I do not advise changing it, since this value strongly influences, for example, std namespace). This means that the size of any structure will always be a multiple of 4 (if the size of the structure was 14, then 2 meaningless bytes will be added to the end and the size will be equal to 16). Therefore, we need to take care of the same value of this parameter on the server and client.
Also, the main error in designing such protocols is inattention to storing multibyte types in memory. It must be remembered that x86 stores them in the form of little-endian (from the youngest to the oldest), and according to the standards of network protocols and for example in SPARC computers it is necessary to store them in the form of big-endian (from the older to the youngest). Thus, we need to know in what order multibyte types will come to us and, if necessary, turn them over. Moreover, if we need high speed, we have a large flow of exchanged data and we cannot get away from rotations (for example, criteria are important when developing a cross-architectural system of distributed computing), then it is necessary to give the function of turning extra half an hour, but write it as fast as possible. In such cases, the standard htonl (), ntohl () may not be in time.
I now refer to tag protocols as fashionable XML-like protocols. Although such protocols are extremely redundant, they are nevertheless easily processed and are completely flexible. Their main problems are redundancy and not very high processing speed.
The main mistake of designing such protocols is the desire to cram everything at once. It is necessary as clearly as possible to formulate the requirements for the protocol and isolate the subset of functionality that is really necessary. Such an approach to design is possible and is not very expandable, but it will allow us to save on processing time. Moreover, with proper design of the parser module, we can reduce the problem of extensibility to a minimum (add a couple of functions and checks to the common code).
To me personally (due to the specialization on speed-demanding and rigidly structured network applications), this approach seems redundant and wasteful.
Perhaps the most interesting type of protocols. Allows you to combine high parsing speed and flexibility.
Packages of this protocol are divided into types. You can also design a type subordination tree (for example, package A can only be included in package B, but cannot go separately). The basis of such packages is a rigidly defined header structural part (for each type has its own) + non-rigid part of the data (the same approach is used in structural data with a variable length of the last parameter in the structure).
Analysis of such protocols, though more difficult than the analysis of structural protocols, but easier and faster than the analysis of purely tagged protocols.
Usually the type of the package is written by the first field in the header part and it is enough for us to read it and call the necessary function that will copy the header into the structure and the data into a piece of memory (usually char * is enough, but in some cases it is more convenient to copy into the array of structures immediately).
The main error in the design is the desire to make "as in the tag protocol" and create a bunch of different types with small headers. This leads to a loss of high parsing speed and negates all the advantages of this type. Thus it is necessary to balance between slowness and slow speed. As practice has shown, in most cases, perfect tagging is the same as breaking up into classes if this protocol were a normal data structure.
PS This post is more a review than a specific implementation. If anyone has a desire to read about some specific implementations (both well-known and designing any others), then write in the comments, I will try to write.
Introduction
Consider the main types of protocols:
- question answer
- structures
- tags
- hybrid tags and structures
Question answer
These protocols are based on communication in small portions of data. The communication protocol is usually strongly typed. As an example, you can bring all known AT-commands for modems.
This type of protocol is the easiest to process (an elementary parsing of a string with isolation of data is required). But to communicate with such a protocol on more or less serious tasks is not very easy. Such protocols are well suited for sending small portions of scalar data types (strings, numbers).
Nevertheless, it is also necessary to design such protocols.
Example
We need to transfer 10 numbers from the client to the server, and receive in response a string or a number (varies from the input data). For this we need: the “handshake” stage, the data transfer stage, the answer receiving stage.
“Manpower” : it is quite enough to send the word “HELLO” back and forth (if we know that there is a possibility to get to another client instead of the server, then we can separate the client and server greetings, for example, “HELLOCL” (from the client) and “HELLOSRV” (from server)). The processing is elementary and comes down to string comparison.
Data transfer : we take the command “SEND x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” as the sending command from the client and “OK SEND” as the server response. The processing is again elementary and comes down to string comparison and calling sscanf ().
Receiving the answer : let's agree that the server sends an “ANSWER STR string ” (if the answer is a string) or an ANSWER NUM number (if the answer is a number). The client responds with the “OK ANSWER” command.
It would seem that everything is simple and clear. BUT. After all, this way we will not be able to understand to which set of numbers the answer sent belongs. To solve this problem is simple. When sending data, we will use the commands: “SEND id NUMS x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” and “OK SEND id ”, where id is the unique identifier for this set of numbers. When responding, the commands will be as follows: “ANSWER id STR string ”, “ANSWER id NUM number ”, “OK ANSWER id ”. Such a protocol will already be exhaustive in most cases.
')
Structures
This type of protocol is the most common. Its basis is hard typing of a portion of the sent data. That is, we predetermine in advance that in all packets at such a displacement and such a length such and such data will lie (the displacement and length of some fields can also be specified in the structure, but mainly the initially specified displacements are used). These protocols are almost all low-level protocols.
The undoubted advantage of such protocols (especially under the condition of rigidly specified displacements and lengths of all fields) is the extreme simplicity of processing. We just need to check the size and execute the memcpy () command into an instance of the structure corresponding to our package.
When designing such protocols, it is necessary to remember about the features of storing structures in C. I mean what is called packing. The fact is that any instance of any structure must be aligned in memory with a certain multiplicity (the multiplicity is set for the whole program alone). By default, the multiplicity is 4 (I do not advise changing it, since this value strongly influences, for example, std namespace). This means that the size of any structure will always be a multiple of 4 (if the size of the structure was 14, then 2 meaningless bytes will be added to the end and the size will be equal to 16). Therefore, we need to take care of the same value of this parameter on the server and client.
Also, the main error in designing such protocols is inattention to storing multibyte types in memory. It must be remembered that x86 stores them in the form of little-endian (from the youngest to the oldest), and according to the standards of network protocols and for example in SPARC computers it is necessary to store them in the form of big-endian (from the older to the youngest). Thus, we need to know in what order multibyte types will come to us and, if necessary, turn them over. Moreover, if we need high speed, we have a large flow of exchanged data and we cannot get away from rotations (for example, criteria are important when developing a cross-architectural system of distributed computing), then it is necessary to give the function of turning extra half an hour, but write it as fast as possible. In such cases, the standard htonl (), ntohl () may not be in time.
Tags
I now refer to tag protocols as fashionable XML-like protocols. Although such protocols are extremely redundant, they are nevertheless easily processed and are completely flexible. Their main problems are redundancy and not very high processing speed.
The main mistake of designing such protocols is the desire to cram everything at once. It is necessary as clearly as possible to formulate the requirements for the protocol and isolate the subset of functionality that is really necessary. Such an approach to design is possible and is not very expandable, but it will allow us to save on processing time. Moreover, with proper design of the parser module, we can reduce the problem of extensibility to a minimum (add a couple of functions and checks to the common code).
To me personally (due to the specialization on speed-demanding and rigidly structured network applications), this approach seems redundant and wasteful.
Tags + structures
Perhaps the most interesting type of protocols. Allows you to combine high parsing speed and flexibility.
Packages of this protocol are divided into types. You can also design a type subordination tree (for example, package A can only be included in package B, but cannot go separately). The basis of such packages is a rigidly defined header structural part (for each type has its own) + non-rigid part of the data (the same approach is used in structural data with a variable length of the last parameter in the structure).
Analysis of such protocols, though more difficult than the analysis of structural protocols, but easier and faster than the analysis of purely tagged protocols.
Usually the type of the package is written by the first field in the header part and it is enough for us to read it and call the necessary function that will copy the header into the structure and the data into a piece of memory (usually char * is enough, but in some cases it is more convenient to copy into the array of structures immediately).
The main error in the design is the desire to make "as in the tag protocol" and create a bunch of different types with small headers. This leads to a loss of high parsing speed and negates all the advantages of this type. Thus it is necessary to balance between slowness and slow speed. As practice has shown, in most cases, perfect tagging is the same as breaking up into classes if this protocol were a normal data structure.
PS This post is more a review than a specific implementation. If anyone has a desire to read about some specific implementations (both well-known and designing any others), then write in the comments, I will try to write.
Source: https://habr.com/ru/post/42940/
All Articles