Transcend Library 4: Introduction
Work with the network in c ++ just, two, three.
Good day dear habravchane, today I want to tell you about my work over the past 4 years. About my creation, about my life. It will be a question of a multifunctional library for working with the network, its history and the reasons for its creation, problems to be solved, as well as the internal structure.
I want to make a reservation right away, maybe my job is a bicycle, but I couldn’t find any analogues that satisfy me either at the time of creation or at the time of switching to new versions.
')
It all started back in 2008, in the summer. We just went through winsock, tried to make our first server.I got a teacher that was unimportant, because nothing was clearly understood. I was a fool and did not understand anything.
I was also extremely upset by the inconvenient api, and immediately wanted to encapsulate the whole thing. I did not know c ++ then, and the first version was just renaming functions (now lost).
More and more, I was moving away from the native winsock scheme, making the communication more transparent. (For example, when calling the connect function, a socket was immediately created, a dns request was made, and the connection was opened). Gradually, it became so convenient that I decided to try to make a game.
However, before the TL2 version, there were no significant improvements, and the “library” itself had, God forbid, 300 lines of code.
With the advent of a real task, a problem arose - transferring several data in one connection (chat, texture, music). It is from this that progress itself began. For half a year I solved this problem, selected the optimal packet splitting, the format of service messages, did tests. As a result, the library successfully coped with the task, with the support of simultaneous data transfer up to 255 "messages" simultaneously, with a maximum size of 4 gigabytes each.
Unfortunately, we never wrote the game due to the leakage of personnel, altruists, after all. Yes, and competitors appeared. Therefore, I was able to leave in the development of the following versions.
The next iteration was the transition to C ++ and the addition of new standards, the use of new features and new features.
Now the library began to work fully in two modes at the same time - the server and the client, allowing you to solve proxy or p2p tasks. The maximum message size has increased to infinity, and the maximum number of simultaneously transmitted up to 2 ^ 32 (with 255 remaining actively transmitted). Added auto-reconnect features, priority appeared to send messages.
The library became asynchronous, with non-blocking sockets, reducing the load on the main thread by several times. (For example, the standard connection opening took me 443 ms in TL2, this number was reduced to 17 ms TL4).
Added encryption, compression, etc., the list can be continued for a long time.
The library allows, regardless of the platform, to do the following:
To the client - open the connection, and transfer any data set from any sources (file, array, other connection) and be sure that no matter how many times the connection is interrupted, the data will be delivered.
To the server - to listen to the port and always, it is intuitive to have information about the number of connections, traffic on each connection, time online, average speed, and so on.
Of course, the example is played out, without error handling, and in general a curve (when trying to convert message to std :: string, the program will be blocked until the arrival of \ 0 or by timeout), but the essence is clear.
This is a very handy thing, allowing you to develop client-server applications without duplicating code.
A little introduction. For each object and each package, a queue of handlers is called (it cannot be changed on the fly for obvious reasons). The handler has two methods - Encode and Decode. Thus, the encryption / decryption code, compression / decompression code is always in one place. This allows you to verify the correctness of the handler - it is enough to call the Test method, the one on the random or provided data will check that after the encryption and decryption, the original version is obtained.
The same is true in your protocol - for the protocol as a whole, there are the same methods that allow you to convert an object into a message and a message into an object.
Everything in the programming world can be represented through a sequence. An array, string, file is all a sequence of bytes. Also about the variable and incoming-outgoing connection.
The stream class system was created in order to abstract from the physical source of the stream and abstract from the transformations actually occurring over information.
Something similar to stl :: stream but with some improvements.
Imagine a simple task, we want to transfer a picture from the disk. To do this, you need to download it, convert bmp-> jpg, encrypt. Now this can be done simply by putting the handlers one by one, and saying that this is an atomic transformation (anologia with a water supply system - the pipes go to butt).
I am sure that from the first time it seems not very clear and convenient, but be sure, after getting acquainted with api it becomes easy.
As you may have guessed, when creating a connection, there is a single logical functionality:
Each byte, when sent, will pass through this handler, and each byte, when received, will also pass through a corresponding handler (pipe :: GetInverseScheme).
The idea is painfully simple, I hope it will be useful to those who need it, and who have not done it yet.
All objects in the queue for sending are shredded by 255 bytes, and are sent by these blocks. Blocks are supplied with an id object. Objects are transmitted cyclically - the first second third first second third.
I note that the blocks are always 255 bytes, and are not supplied with size. If suddenly the unit is smaller, it will be adjusted through the service messages.
If suddenly the next portion of the object is “not ready” (no data), then it moves to the inactive queue, instead of one ready. All this is regulated through system messages.
They carry several functions. The transferring party reports active-inactive rotations, warns of the approaching end of the object, reports the addition of objects, the transfer of "low-level" exceptions, the transfer of the pipe format for the object and the most interesting thing is synchronization.
For proper operation it must be turned on two ends. It was because of this task that we had to implement the priority of objects, it is necessary that these service messages be added to the queue at the time of creation.
Hosts exchange their timestamp with milliseconds, thus getting ping. After setting the average value, they begin to share it too. As a result, both cars have a response time.
The most interesting begins in a server solution. Players with ping more than a couple of hundred will understand me. The enemy for a few milliseconds learns something important earlier, and this is all decided.
And now imagine the games where events come on all machines at the same time, and it is done simply. When sending a “concurrent” message, the connections are sorted in order of delays, and the actual sending of data occurs so that the estimated arrival time is equal, taking into account the stability of the connection (error less than 10ms for wired solutions).
The fact is that the library is not complete. For example, not all the planned connection parameters are supported, I would like out-of-the-box encryption algorithms, check sum, and so on. In general, the work is still underway.
Here I didn’t appear to be promoted, but to find out if there are analogues (I would like tosteal a couple of ideas to look like people’s), whether anyone needs this (it makes sense to enter the market) and find out what you wouldn’t have enough the system.
Please forgive me for my presentation, mistakes and inaccuracies. I was unable to add the “I am PR” tag due to low karma. And indeed in vain, I guess it is.
UPD: Continued
network n = OpenNetwork(); connection c = n.OpenConnection("habrahabr.ru", 80); html_request h("GET", "/"); c << h; c.WaitForMessage(); n.Response(); std::cout << c; Good day dear habravchane, today I want to tell you about my work over the past 4 years. About my creation, about my life. It will be a question of a multifunctional library for working with the network, its history and the reasons for its creation, problems to be solved, as well as the internal structure.
I want to make a reservation right away, maybe my job is a bicycle, but I couldn’t find any analogues that satisfy me either at the time of creation or at the time of switching to new versions.
')
Story
Start
It all started back in 2008, in the summer. We just went through winsock, tried to make our first server.
I was also extremely upset by the inconvenient api, and immediately wanted to encapsulate the whole thing. I did not know c ++ then, and the first version was just renaming functions (now lost).
More and more, I was moving away from the native winsock scheme, making the communication more transparent. (For example, when calling the connect function, a socket was immediately created, a dns request was made, and the connection was opened). Gradually, it became so convenient that I decided to try to make a game.
Ancient legend
However, before the TL2 version, there were no significant improvements, and the “library” itself had, God forbid, 300 lines of code.
With the advent of a real task, a problem arose - transferring several data in one connection (chat, texture, music). It is from this that progress itself began. For half a year I solved this problem, selected the optimal packet splitting, the format of service messages, did tests. As a result, the library successfully coped with the task, with the support of simultaneous data transfer up to 255 "messages" simultaneously, with a maximum size of 4 gigabytes each.
Unfortunately, we never wrote the game due to the leakage of personnel, altruists, after all. Yes, and competitors appeared. Therefore, I was able to leave in the development of the following versions.
Tl4
The next iteration was the transition to C ++ and the addition of new standards, the use of new features and new features.
Now the library began to work fully in two modes at the same time - the server and the client, allowing you to solve proxy or p2p tasks. The maximum message size has increased to infinity, and the maximum number of simultaneously transmitted up to 2 ^ 32 (with 255 remaining actively transmitted). Added auto-reconnect features, priority appeared to send messages.
The library became asynchronous, with non-blocking sockets, reducing the load on the main thread by several times. (For example, the standard connection opening took me 443 ms in TL2, this number was reduced to 17 ms TL4).
Added encryption, compression, etc., the list can be continued for a long time.
Main tasks to be solved
The library allows, regardless of the platform, to do the following:
To the client - open the connection, and transfer any data set from any sources (file, array, other connection) and be sure that no matter how many times the connection is interrupted, the data will be delivered.
To the server - to listen to the port and always, it is intuitive to have information about the number of connections, traffic on each connection, time online, average speed, and so on.
Example "PROXY"
#include <tl4/def.h> using namespace ax::tl4; bool proxy( connection &c, message &m ) { std::string s; s << m; // host:pass, connection nc = c.GetNetwork().OpenConnection(s); nc << s; nc << c; nc >> c; return true; } void main() { network n = OpenNetwork(); port p = n.OpenPort(8080); p.NonTL4Compatible(true); p.OnOpenConnection(proxy); while (1) n.Response(); } Of course, the example is played out, without error handling, and in general a curve (when trying to convert message to std :: string, the program will be blocked until the arrival of \ 0 or by timeout), but the essence is clear.
Opportunities
- No protocol restrictions
- Support encryption, compression
- The most accurate synchronization
- Making your handlers (encryption) at the level of the message package. Fully variable packet construction pipe.
- Combining connections into “virtual networks” (groups processed in “one go”). All communication between networks is thread safe.
- Transfer up to 4 billion objects simultaneously through one connection.
- Adjustable connection parameters, software speed limit and the like.
- Custom event handlers for each connection or group.
- Priorities for the transfer of objects, the full control of "what exactly will be transferred now."
- The ability to transfer not fully available objects - for example, a torrent, a stream of Internet radio. No size limit.
- Ability to process "new object", "new piece of object", "completion of downloading object", "every N bytes", "every N seconds".
- Development of the protocol in one place.
Protocol in one place
This is a very handy thing, allowing you to develop client-server applications without duplicating code.
A little introduction. For each object and each package, a queue of handlers is called (it cannot be changed on the fly for obvious reasons). The handler has two methods - Encode and Decode. Thus, the encryption / decryption code, compression / decompression code is always in one place. This allows you to verify the correctness of the handler - it is enough to call the Test method, the one on the random or provided data will check that after the encryption and decryption, the original version is obtained.
The same is true in your protocol - for the protocol as a whole, there are the same methods that allow you to convert an object into a message and a message into an object.
How it works
Stream
Everything in the programming world can be represented through a sequence. An array, string, file is all a sequence of bytes. Also about the variable and incoming-outgoing connection.
The stream class system was created in order to abstract from the physical source of the stream and abstract from the transformations actually occurring over information.
Something similar to stl :: stream but with some improvements.
Pipe
Imagine a simple task, we want to transfer a picture from the disk. To do this, you need to download it, convert bmp-> jpg, encrypt. Now this can be done simply by putting the handlers one by one, and saying that this is an atomic transformation (anologia with a water supply system - the pipes go to butt).
pipe *ImageTransferPipe( std::string filename ) { pipe_constructor c; c.add(new BMP2JPG()); c.add(new BlowFishCrypt()); pipe p(c); return p.GetScheme().Source(new read_file_stream(filename)); } I am sure that from the first time it seems not very clear and convenient, but be sure, after getting acquainted with api it becomes easy.
Out-of-box encryption
As you may have guessed, when creating a connection, there is a single logical functionality:
connection::SetHandler( pipe * ) Each byte, when sent, will pass through this handler, and each byte, when received, will also pass through a corresponding handler (pipe :: GetInverseScheme).
Transfer of several objects simultaneously
The idea is painfully simple, I hope it will be useful to those who need it, and who have not done it yet.
All objects in the queue for sending are shredded by 255 bytes, and are sent by these blocks. Blocks are supplied with an id object. Objects are transmitted cyclically - the first second third first second third.
I note that the blocks are always 255 bytes, and are not supplied with size. If suddenly the unit is smaller, it will be adjusted through the service messages.
If suddenly the next portion of the object is “not ready” (no data), then it moves to the inactive queue, instead of one ready. All this is regulated through system messages.
System messages
They carry several functions. The transferring party reports active-inactive rotations, warns of the approaching end of the object, reports the addition of objects, the transfer of "low-level" exceptions, the transfer of the pipe format for the object and the most interesting thing is synchronization.
Sync machines
For proper operation it must be turned on two ends. It was because of this task that we had to implement the priority of objects, it is necessary that these service messages be added to the queue at the time of creation.
Hosts exchange their timestamp with milliseconds, thus getting ping. After setting the average value, they begin to share it too. As a result, both cars have a response time.
The most interesting begins in a server solution. Players with ping more than a couple of hundred will understand me. The enemy for a few milliseconds learns something important earlier, and this is all decided.
And now imagine the games where events come on all machines at the same time, and it is done simply. When sending a “concurrent” message, the connections are sorted in order of delays, and the actual sending of data occurs so that the estimated arrival time is equal, taking into account the stability of the connection (error less than 10ms for wired solutions).
Why this post?
The fact is that the library is not complete. For example, not all the planned connection parameters are supported, I would like out-of-the-box encryption algorithms, check sum, and so on. In general, the work is still underway.
Here I didn’t appear to be promoted, but to find out if there are analogues (I would like to
Public apologies
Please forgive me for my presentation, mistakes and inaccuracies. I was unable to add the “I am PR” tag due to low karma. And indeed in vain, I guess it is.
UPD: Continued
Source: https://habr.com/ru/post/151812/
All Articles