About data transfer protocols
In this article, I want to share my personal, non-academic, experience in working with various private data transfer protocols (mainly at the application-session level).
I often have to interface with specialized software (and hardware, although eventually they interface with a firmware line), the manufacturer of each of which provides its own data exchange protocol.
What properties and features does agood, suitable, competent, high-quality protocol have?
Ideally, the protocol should be abstracted from a lower level of interaction, whether it be transmission via TCP, UDP, serial port, USB, Bluetooth, digital radio, or even pigeon mail . And we must bear in mind that not all of them guarantee the delivery and / or reliability of the transmitted data.
')
A small disclaimer: speaking of the reliability of the data, I mean their non-distortion due to interference and other errors in the transmission medium. In this article I will not touch upon the topics of the layer of technologies related to security in IT. Suppose that our Alice and Bob can trust each other, and no Eve can stop them . (For example, for colleagues, the security issue is solved by including all geographically separated interaction participants in a well-protected VPN, which in turn does not have access to the outside)
Most protocols implemented the question-answer scheme. This can be represented as a conversation in which you respond verbally to each remark of your interlocutor, and in the same meaning. In this way, the participants in the interaction achieve confidence that their messages are transmitted and adequately received. However, this scheme is permissible and effective not for all tasks: in cases where the delay in communication must be minimized, or the answer to each of the numerous replicas is considered redundant (for example, for debug messages), the Start-Stop scheme is implemented. When you receive a message on the "Start" your interlocutor begins to pour in you a stream of replicas, and stops only at the word "Stop". Messages sent in the stream usually have an incremental sequence number, and if one of them had problems with the processing of the message stream, one of them was skipped, it can be re-requested separately by this very number.
All protocols can be divided into two groups, (according to the presentation of data): symbolic and binary .
The character protocols I met were based either on XML or on JSON strings. From their merits we can mention the simpler debugging of the interaction (due to their readability), the ease of implementation (the availability of ready-made parsers), and the notorious universality.
Now about the shortcomings. Obviously, such protocols are extremely redundant, a tiny fraction of useful information is floating in a massive, inefficient wrapper. When transmitting any numerical information, you have to convert them into string representation and back. The sore spot is the transfer of binary data (and it’s good that you can do without them, but in some cases this is not possible). The compilers of protocols are usually unscrewed using Base64 , or even simply by passing a binary string in its hex representation, two characters per byte.
I would also like to note that the full specification of the same XML is extremely extensive, and standard parsers, with all their full capabilities, are rather cumbersome and slow, so the practice is common when a department or office eventually writes and uses its own parser.
Of course, for certain tasks , symbolic protocols are, if not the most effective, then at least a quite acceptable option, but we go further.
Now binary protocols. Immediately we must recall the Gulliver wars of blunt and pointed . Personally, I sympathize with big-endian, because I don’t consider implicit little-endian typing to be “something good”, and in my development environment big-endian is native.
Binary protocols (not all, but those that I attribute to literate) can be divided into two levels: the container level and the data level. The first level is responsible for the integrity and reliability of data transmission, as well as for the availability of message detection in the byte stream, and, of course, for storing the data level messages in itself. The second level should contain information for which all network interaction was started, in a format convenient for processing. Its structure mainly depends on the tasks to be solved, but there are general recommendations for it (about which below).
Message sizes (discrete packet bytes that can be processed independently of the previous and subsequent received data) are fixed and variable . It is clear that with a fixed message size everything is simpler - it is subtracted, starting with the header (about it later), a certain number of bytes and sent for processing. Often, for flexibility, the originators of such protocols include a fixed-size region (sometimes up to 80% of the total) reserved for modifications to the current protocol. In my opinion, this is not the most effective way to ensure flexibility, but there is still some redundancy.
Consider messages of variable length.
Here you can talk in more detail about the indispensable attribute of a binary message in any protocol - about the header (This is the above-mentioned level of the container).
Typically, headers begin with a constant part, which allows, with a certain probability, to detect the beginning of a message in a continuous byte stream. Obviously, there is a risk of such a constant appearing in an arbitrary stream of bytes, and although the increase in volume reduces this risk (I have met constants of the form 0123456789VASIA9876543210), it is more expedient to use checksum- based checks .
The constant is usually followed by the version number of the protocol, which lets us know in what format further reading should take place (and whether we even have the opportunity to process this message — suddenly this version is unknown to us). The next important part of the header: information about the contents of the container. Specify the type of content (in fact, the same protocol version number for the data layer), its length and checksum. Having this information, it is already possible without problems and fears to read the contents and begin to analyze it.
But not right away! The title should contain the checksum of itself ( excluding from the calculation the checksum itself, of course ) - this is the only way we can be sure that we considered not just a rubbish, but a valid title followed by the data intended for us. Did not match the checksum? We'll have to look for the next beginning of the new heading further downstream ...
Imagine that we have reached the stage that we finally received an undistorted data-level message. Its structure depends on the task area of ​​the system in which your network exchange is implemented, however, in general, a message also has its own header containing information about the type of message. You can distinguish between the general specifics of the message (for example, Set Request, Affirmative Answer to Set, Negative Answer to Set, Get Request, Get Answer, Stream Message) and the specific application area of ​​the message . I will try to give an example from the ceiling:
Request Type: Request Set (0x01)
Identifier of the addressee module of the message: PowerSupplyModule (0x0A)
Message group identifier: UPS Management (0x02)
Message Type ID: Reboot (0x01)
Further, the message body may contain information about the address of the UPS, which the Power Management Module should restart, after how many seconds, and so on.
We expect to receive a reply to this message with the request type “Affirmative Answer” and the subsequent 0x0A0201 in the header.
Of course, such a detailed description of the type of message may be redundant when the interconnection does not provide for a large number of commands, so it is necessary to form the structure of the message based on the requirements of the TOR.
It will also be useful if a message with a “Negative Response” contains an error code, due to which the command could not be answered in the affirmative.
Finishing my story, I will add that the topic of application interaction is very extensive and sometimes holivorn (which in fact means that it does not have a “silver bullet” technology), and I note that the views that I present are just a compilation of work experience with domestic and foreign colleagues. Thanks for attention!
upd.
I had the pleasure to talk with the critic of my article, and now I come to the realization that I have covered the issue with my own, so to speak, “baytolyubskaya” point of view. Of course, since there is a course for the versatility of processing storage and data transfer, then in such a key, symbolic protocols (primarily speaking about XML) can give odds to any other solutions. But regarding their widespread use, let me quote Wirth:
The tool must match the task. If the tool does not match the task, you need to come up with a new one that would fit it, and not try to adapt an existing one.
I often have to interface with specialized software (and hardware, although eventually they interface with a firmware line), the manufacturer of each of which provides its own data exchange protocol.
What properties and features does a
Ideally, the protocol should be abstracted from a lower level of interaction, whether it be transmission via TCP, UDP, serial port, USB, Bluetooth, digital radio, or even pigeon mail . And we must bear in mind that not all of them guarantee the delivery and / or reliability of the transmitted data.
')
A small disclaimer: speaking of the reliability of the data, I mean their non-distortion due to interference and other errors in the transmission medium. In this article I will not touch upon the topics of the layer of technologies related to security in IT. Suppose that our Alice and Bob can trust each other, and no Eve can stop them . (For example, for colleagues, the security issue is solved by including all geographically separated interaction participants in a well-protected VPN, which in turn does not have access to the outside)
Most protocols implemented the question-answer scheme. This can be represented as a conversation in which you respond verbally to each remark of your interlocutor, and in the same meaning. In this way, the participants in the interaction achieve confidence that their messages are transmitted and adequately received. However, this scheme is permissible and effective not for all tasks: in cases where the delay in communication must be minimized, or the answer to each of the numerous replicas is considered redundant (for example, for debug messages), the Start-Stop scheme is implemented. When you receive a message on the "Start" your interlocutor begins to pour in you a stream of replicas, and stops only at the word "Stop". Messages sent in the stream usually have an incremental sequence number, and if one of them had problems with the processing of the message stream, one of them was skipped, it can be re-requested separately by this very number.
All protocols can be divided into two groups, (according to the presentation of data): symbolic and binary .
The character protocols I met were based either on XML or on JSON strings. From their merits we can mention the simpler debugging of the interaction (due to their readability), the ease of implementation (the availability of ready-made parsers), and the notorious universality.
Now about the shortcomings. Obviously, such protocols are extremely redundant, a tiny fraction of useful information is floating in a massive, inefficient wrapper. When transmitting any numerical information, you have to convert them into string representation and back. The sore spot is the transfer of binary data (and it’s good that you can do without them, but in some cases this is not possible). The compilers of protocols are usually unscrewed using Base64 , or even simply by passing a binary string in its hex representation, two characters per byte.
I would also like to note that the full specification of the same XML is extremely extensive, and standard parsers, with all their full capabilities, are rather cumbersome and slow, so the practice is common when a department or office eventually writes and uses its own parser.
Of course, for certain tasks , symbolic protocols are, if not the most effective, then at least a quite acceptable option, but we go further.
Now binary protocols. Immediately we must recall the Gulliver wars of blunt and pointed . Personally, I sympathize with big-endian, because I don’t consider implicit little-endian typing to be “something good”, and in my development environment big-endian is native.
Binary protocols (not all, but those that I attribute to literate) can be divided into two levels: the container level and the data level. The first level is responsible for the integrity and reliability of data transmission, as well as for the availability of message detection in the byte stream, and, of course, for storing the data level messages in itself. The second level should contain information for which all network interaction was started, in a format convenient for processing. Its structure mainly depends on the tasks to be solved, but there are general recommendations for it (about which below).
Message sizes (discrete packet bytes that can be processed independently of the previous and subsequent received data) are fixed and variable . It is clear that with a fixed message size everything is simpler - it is subtracted, starting with the header (about it later), a certain number of bytes and sent for processing. Often, for flexibility, the originators of such protocols include a fixed-size region (sometimes up to 80% of the total) reserved for modifications to the current protocol. In my opinion, this is not the most effective way to ensure flexibility, but there is still some redundancy.
Consider messages of variable length.
Here you can talk in more detail about the indispensable attribute of a binary message in any protocol - about the header (This is the above-mentioned level of the container).
Typically, headers begin with a constant part, which allows, with a certain probability, to detect the beginning of a message in a continuous byte stream. Obviously, there is a risk of such a constant appearing in an arbitrary stream of bytes, and although the increase in volume reduces this risk (I have met constants of the form 0123456789VASIA9876543210), it is more expedient to use checksum- based checks .
The constant is usually followed by the version number of the protocol, which lets us know in what format further reading should take place (and whether we even have the opportunity to process this message — suddenly this version is unknown to us). The next important part of the header: information about the contents of the container. Specify the type of content (in fact, the same protocol version number for the data layer), its length and checksum. Having this information, it is already possible without problems and fears to read the contents and begin to analyze it.
But not right away! The title should contain the checksum of itself ( excluding from the calculation the checksum itself, of course ) - this is the only way we can be sure that we considered not just a rubbish, but a valid title followed by the data intended for us. Did not match the checksum? We'll have to look for the next beginning of the new heading further downstream ...
Imagine that we have reached the stage that we finally received an undistorted data-level message. Its structure depends on the task area of ​​the system in which your network exchange is implemented, however, in general, a message also has its own header containing information about the type of message. You can distinguish between the general specifics of the message (for example, Set Request, Affirmative Answer to Set, Negative Answer to Set, Get Request, Get Answer, Stream Message) and the specific application area of ​​the message . I will try to give an example from the ceiling:
Request Type: Request Set (0x01)
Identifier of the addressee module of the message: PowerSupplyModule (0x0A)
Message group identifier: UPS Management (0x02)
Message Type ID: Reboot (0x01)
Further, the message body may contain information about the address of the UPS, which the Power Management Module should restart, after how many seconds, and so on.
We expect to receive a reply to this message with the request type “Affirmative Answer” and the subsequent 0x0A0201 in the header.
Of course, such a detailed description of the type of message may be redundant when the interconnection does not provide for a large number of commands, so it is necessary to form the structure of the message based on the requirements of the TOR.
It will also be useful if a message with a “Negative Response” contains an error code, due to which the command could not be answered in the affirmative.
Finishing my story, I will add that the topic of application interaction is very extensive and sometimes holivorn (which in fact means that it does not have a “silver bullet” technology), and I note that the views that I present are just a compilation of work experience with domestic and foreign colleagues. Thanks for attention!
upd.
I had the pleasure to talk with the critic of my article, and now I come to the realization that I have covered the issue with my own, so to speak, “baytolyubskaya” point of view. Of course, since there is a course for the versatility of processing storage and data transfer, then in such a key, symbolic protocols (primarily speaking about XML) can give odds to any other solutions. But regarding their widespread use, let me quote Wirth:
The tool must match the task. If the tool does not match the task, you need to come up with a new one that would fit it, and not try to adapt an existing one.
Source: https://habr.com/ru/post/138533/
All Articles