Zoo communication protocols for GPS trackers (part 1)
The purpose of this article is to acquaint the reader with the various application layer network protocols that are used in GPS trackers. Almost every manufacturer of equipment for GPS monitoring considers it his duty to invent a new protocol for interaction between the device and the monitoring server. Often, when designing these engineers, they resort to sophisticated solutions to save traffic or simplify the lives of device firmware developers.
At the transport level, in most cases, TCP and sometimes UDP is used. The application layer protocols themselves can be divided into two large groups: text and binary. In text protocols, information is transmitted in the form of ASCII codes, and after conversion to a string, it is often possible to parse the data even without documentation. Text protocols contain a sufficiently large amount of redundant information, so some manufacturers use binary protocols in which data is transmitted in big-endian or little-endian binary format.
In the case of using TCP before decoding messages, it is necessary to split the data stream into separate frames. The standard for text protocols is the splitting of messages using one or more special characters. The most common character used as a delimiter is a newline character (ASCII code 0x0A). Binary protocol messages usually contain a fixed-length header that includes a packet length field indicating the size of the data, or the size of the entire message.
Now consider some of the most interesting non-standard cases.
')
In the old version of the protocol of the company Shenzhen Xexun Technology Co, after the text data, a checksum is transmitted in binary format. The problem is also complicated by the fact that there is no separator character. After being converted into text, the message looks something like this (binary data is indicated by a symbol):
To extract the data in this case, we had to use textual redundancy text and look for tags (longer than 2 characters): GPRMC and imei.
There are a lot of such protocols, but in most cases the decoding logic is quite simple. For example, all messages can be of a fixed length, or the length can be uniquely determined by the type of message (the type of message is contained in a fixed header). In this case, the Taiwanese company Orion Technology, which describes its protocol using the following grammar:
Only a small part of the document is shown, from which it is clear that to determine the length of a frame with GPS data (TRACKING), you must first look at the type of the entire message, then take the number of different records (USER-LOG) from the data header, and finally read the header of each record and add up the lengths.
Ulbotech decided that using a special code (0xF8) to separate packages would be a good idea. Apparently, realizing later that this code can also sometimes be found in binary data, they decided to replace each of its entries with a code of two bytes (0xF7 0x0F). The problem is that this code can also occur in binary data, and therefore the code 0xF7 is replaced with the next code of two bytes (0xF7 0x00).
For example, if the tracker sends the following sequence:
then after decoding we get (separator deleted):
Some protocols used in GPS trackers contain both binary and text messages. There are a lot of examples; in most cases, location data is transmitted in text form, and in a binary form, keepalive messages are transmitted to indicate the active connection between the server and the client.
An example would be a protocol called WondeX from the Taiwanese company Wonde Proud. The keepalive message looks like this (little-endian):
In this case, the first byte of the message (code 0xD0 can not be found in the text data) can determine the type and further, depending on the type, determine the frame size. In the case of a binary message, the WondeX frame has a fixed length, and text messages use a line feed as a separator.
The definition of frame length is just the beginning. After that, you need to decipher the data itself, and here you can find even more innovative solutions. At first I wanted to include everything in one article, but after writing the part about frame decoders, I decided that this information is enough for one article. I would be happy to hear feedback and find out whether it makes sense to write a sequel.
All the information in the article was accumulated in the course of work on the GPS monitoring server. The project is Open Source, and if anyone is interested, the source code with the implementation of protocols can be found on GitHub .
Also, someone may be interested in a collection of protocols for GPS-trackers .
At the transport level, in most cases, TCP and sometimes UDP is used. The application layer protocols themselves can be divided into two large groups: text and binary. In text protocols, information is transmitted in the form of ASCII codes, and after conversion to a string, it is often possible to parse the data even without documentation. Text protocols contain a sufficiently large amount of redundant information, so some manufacturers use binary protocols in which data is transmitted in big-endian or little-endian binary format.
Frame decoders
In the case of using TCP before decoding messages, it is necessary to split the data stream into separate frames. The standard for text protocols is the splitting of messages using one or more special characters. The most common character used as a delimiter is a newline character (ASCII code 0x0A). Binary protocol messages usually contain a fixed-length header that includes a packet length field indicating the size of the data, or the size of the entire message.
Now consider some of the most interesting non-standard cases.
')
Binary data in text protocol
In the old version of the protocol of the company Shenzhen Xexun Technology Co, after the text data, a checksum is transmitted in binary format. The problem is also complicated by the fact that there is no separator character. After being converted into text, the message looks something like this (binary data is indicated by a symbol):
111111120009,+1234,GPRMC,204530.4,A,6000.000,N,01000.6288,E,0.0,0.00,230713,0.0,E,A*3C,F,imei:123456789012345,00,,F:3.88V,0,125,,262,01,224CE1,379B To extract the data in this case, we had to use textual redundancy text and look for tags (longer than 2 characters): GPRMC and imei.
Binary messages without length in title
There are a lot of such protocols, but in most cases the decoding logic is quite simple. For example, all messages can be of a fixed length, or the length can be uniquely determined by the type of message (the type of message is contained in a fixed header). In this case, the Taiwanese company Orion Technology, which describes its protocol using the following grammar:
GPRS-PACKET := PKT-HEADER PKT-BODY PKT-NO PKT-CRC PKT-BODY := (USERLOG-DATA | SYSLOG-DATA) ... USERLOG-DATA := USERLOG-HDR USERLOG-COMMON USER-LOGS USER-LOGS := 1*N (USER-LOG) USER-LOG := (SPEED-ALERT | TRACKING | ...) ... SYSLOG-DATA:= SYSLOG-HDR UNIT-ID SYSLOG-LEN ... ... Only a small part of the document is shown, from which it is clear that to determine the length of a frame with GPS data (TRACKING), you must first look at the type of the entire message, then take the number of different records (USER-LOG) from the data header, and finally read the header of each record and add up the lengths.
Binary protocol with delimiter character
Ulbotech decided that using a special code (0xF8) to separate packages would be a good idea. Apparently, realizing later that this code can also sometimes be found in binary data, they decided to replace each of its entries with a code of two bytes (0xF7 0x0F). The problem is that this code can also occur in binary data, and therefore the code 0xF7 is replaced with the next code of two bytes (0xF7 0x00).
For example, if the tracker sends the following sequence:
0xF8 0x00 0xF7 0x00 0xFF 0xF7 0x0F 0xF8 then after decoding we get (separator deleted):
0x00 0xF7 0xFF 0xF8 Mixed protocols
Some protocols used in GPS trackers contain both binary and text messages. There are a lot of examples; in most cases, location data is transmitted in text form, and in a binary form, keepalive messages are transmitted to indicate the active connection between the server and the client.
An example would be a protocol called WondeX from the Taiwanese company Wonde Proud. The keepalive message looks like this (little-endian):
0xD0 0xD7 0x1A 0x01 0xC7 0x54 0x44 0x3C In this case, the first byte of the message (code 0xD0 can not be found in the text data) can determine the type and further, depending on the type, determine the frame size. In the case of a binary message, the WondeX frame has a fixed length, and text messages use a line feed as a separator.
Conclusion
The definition of frame length is just the beginning. After that, you need to decipher the data itself, and here you can find even more innovative solutions. At first I wanted to include everything in one article, but after writing the part about frame decoders, I decided that this information is enough for one article. I would be happy to hear feedback and find out whether it makes sense to write a sequel.
All the information in the article was accumulated in the course of work on the GPS monitoring server. The project is Open Source, and if anyone is interested, the source code with the implementation of protocols can be found on GitHub .
Also, someone may be interested in a collection of protocols for GPS-trackers .
Source: https://habr.com/ru/post/252715/
All Articles