Remote execution of system commands on request via sockets in Python 3. Part 2. Data transfer protocol
In the previous article, I explained how to create a server and client in Python 3 using embedded sockets. But this application had many flaws, which I will try to correct in this and subsequent articles.
So what are the disadvantages of our application?
Today I will tell how to solve the first problem, and at the same time a little about TCP.
We used the bare protocol TCP to transfer data between the server and the client. TCP is a streaming protocol; it transmits data in a sequential set of bytes. Transmitting a command with arguments over the network in the first version of our application, we read only 1024 bytes of data from the received packet. But what if the data does not fit in 1024 bytes? There is only one way out - to split the data into several packages on one host and “glue” them into one piece when received on another host. But how do you know when one team ends (with its arguments) and another begins? To do this, we need to know how long the entire transmitted message is.
')
Since we cannot find out the length of the message in advance, we will have to transmit it in one of the packets. Of course, it is better to do this at the very beginning of the first package. Having allocated only 4 bytes for storing the message length, we will be able to send a message of over 4 billion characters! The message length is information about it, that is, part of the header, the header of our protocol. What protocol do you ask? If you believe Wikipedia, then
We agreed that we will transmit data in several packets via TCP, and at the beginning of the data of the first packet the length of the entire message in bytes will be stored. So we developed our simple protocol! It must be remembered that our protocol is based on TCP, which means it has the same features as the last one.
We described our protocol, it is time to develop it!
The sendall socket method in Python independently breaks the data into packets and sends them to the server. Here you can not bathe. To transfer the length we translate it into a C struct of type unsigned int (4 bytes) using the built-in library struct .
The '> I' parameter passed to the struct.pack function asks to transfer the second parameter to the unsigned int ( I ) type in reverse byte order ( > ). Consider this later.
In the recv_packets function , we read already received data until we receive a part of the message of the required length. If the socket recv method returns nothing, then we could not receive the message completely. In this case, too, nothing is returned, and then nothing returns the recv function of our protocol.
Now we can use our protocol written on top of another protocol - TCP. At the same time, I will talk about what is happening at this moment under the hood.
To get started, go to the directory with the newly created protocol.py file and run the Python 3 interpreter (usually the python3 command) in two terminals, command line interpreters, or development environment. In both, enter the following commands in turn.
The first line imports the socket library and our mini-protocol, the second one creates a socket with the TCP / IPv4 protocol. AF_INET — pair (domain / IPv4, port), SOCK_STREAM — streaming connection type on which the TCP protocol is based.
The next two lines are entered in the first terminal.
At this point, the operating system takes up the address localhost: 65043 for our application and our application begins to listen to it. If you have not yet understood, a socket is a software interface that is created by the operating system.
We join the server through the second terminal.
At this point, the following happens. The client is sending a small package. My first packet is 74 bytes long. (The length is not always the same, I quote it so that it is approximately clear what is being sent at the moment of establishing a connection.) They can be expanded as follows: 8 bytes - two Ethernet addresses, 20 bytes - an IP header, and 46 bytes - a TCP header . There are two bits in the packet from the TCP header — syn and ack . In the first packet, syn = 1, ack = 0 . After that, the server sends us a packet of the same length (74 bytes), acknowledging receipt and allowing to connect, with syn = 1, ack = 1 . Then the client sends the third packet, but not 74, but 66 bytes in length. In the third packet, syn = 0, ack = 1 . This package finally establishes the connection and now we can receive and receive packets. It looks like a successful connection. If you would like to study TCP and other possible cases in more detail, you can read about it, for example, in the TANENTBAUM book “Computer Networks”.
We read information about the client. At this point, no packets are transmitted, we just take the data already recorded.
In the second terminal, enter
With this line we send a message to the server using our mini-protocol.
At this moment we send a package with the following content to the server (excluding headers):
00: 00: 00: 11 - 0x11 , or 17 in decimal notation, is the length of the transmitted message (not the data, but the message, since the data, in this case, is 4 bytes of length + message).
48: 65: 6c: 6c: 6f: 2c: 20: 6c: 6f: 63: 61: 6c: 68: 6f: 73: 74: 21 - transmitted string Hello, localhost!
The server responds with another packet confirming receipt.
Finally, we read and decode the message in the first terminal, the client:
You can break the connection by typing in the second terminal
The client sends the server a packet with the fin bit set, saying that it has no more data to send and wants to break the connection. In response, the server also sends a packet with the fin bit = 1 .
The same command in the first terminal will stop the server and it will stop listening.
After all we have done, we can easily transfer messages between client and server applications. The changes are not very different, so I did not mention the code listings in the article, but put the project on GitHub . In the next article I will talk about security. If you are interested in something else that you would like to hear about in the following articles, then write about it in the comments.
So what are the disadvantages of our application?
- A single packet is sent, the length of which cannot exceed a predetermined limit of 1 KB.
- An application without verification passes the argument passed from the network to the shell (URL).
- Lack of functionality. We cannot, for example, download all images from Habr, or download a separate hub.
Today I will tell how to solve the first problem, and at the same time a little about TCP.
Protocol Description
We used the bare protocol TCP to transfer data between the server and the client. TCP is a streaming protocol; it transmits data in a sequential set of bytes. Transmitting a command with arguments over the network in the first version of our application, we read only 1024 bytes of data from the received packet. But what if the data does not fit in 1024 bytes? There is only one way out - to split the data into several packages on one host and “glue” them into one piece when received on another host. But how do you know when one team ends (with its arguments) and another begins? To do this, we need to know how long the entire transmitted message is.
')
Since we cannot find out the length of the message in advance, we will have to transmit it in one of the packets. Of course, it is better to do this at the very beginning of the first package. Having allocated only 4 bytes for storing the message length, we will be able to send a message of over 4 billion characters! The message length is information about it, that is, part of the header, the header of our protocol. What protocol do you ask? If you believe Wikipedia, then
A data transfer protocol is a set of logical-level interface agreements that define the exchange of data between different programs.
We agreed that we will transmit data in several packets via TCP, and at the beginning of the data of the first packet the length of the entire message in bytes will be stored. So we developed our simple protocol! It must be remembered that our protocol is based on TCP, which means it has the same features as the last one.
Protocol development
We described our protocol, it is time to develop it!
The sendall socket method in Python independently breaks the data into packets and sends them to the server. Here you can not bathe. To transfer the length we translate it into a C struct of type unsigned int (4 bytes) using the built-in library struct .
import struct # def send(connection, data): # data = bytes(data, "utf8") # C struct unsigned int 4 , . 4 , , . connection.sendall(struct.pack('>I', len(data)) + data) The '> I' parameter passed to the struct.pack function asks to transfer the second parameter to the unsigned int ( I ) type in reverse byte order ( > ). Consider this later.
# def recv_packets(connection, n): piece = b'' # <b>n</b> while len(piece) < n: # , <b>n</b> packet = connection.recv(n - len(piece)) # , if not packet: return None piece += packet return piece # def recv(connection): # , length_data = recv_packets(connection, 4) # , if not length_data: return None # data_len = struct.unpack('>I', length_data)[0] # , UTF-8. "utf8", "utf-8", "UTF8", "UTF-8", . return recv_packets(connection, data_len).decode("utf8")
In the recv_packets function , we read already received data until we receive a part of the message of the required length. If the socket recv method returns nothing, then we could not receive the message completely. In this case, too, nothing is returned, and then nothing returns the recv function of our protocol.
Protocol use
Now we can use our protocol written on top of another protocol - TCP. At the same time, I will talk about what is happening at this moment under the hood.
To get started, go to the directory with the newly created protocol.py file and run the Python 3 interpreter (usually the python3 command) in two terminals, command line interpreters, or development environment. In both, enter the following commands in turn.
>>> import socket, protocol >>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
The first line imports the socket library and our mini-protocol, the second one creates a socket with the TCP / IPv4 protocol. AF_INET — pair (domain / IPv4, port), SOCK_STREAM — streaming connection type on which the TCP protocol is based.
The next two lines are entered in the first terminal.
>>> sock.bind(("localhost", 65043)) >>> sock.listen(True)
At this point, the operating system takes up the address localhost: 65043 for our application and our application begins to listen to it. If you have not yet understood, a socket is a software interface that is created by the operating system.
We join the server through the second terminal.
>>> sock.connect(("localhost", 65043))
At this point, the following happens. The client is sending a small package. My first packet is 74 bytes long. (The length is not always the same, I quote it so that it is approximately clear what is being sent at the moment of establishing a connection.) They can be expanded as follows: 8 bytes - two Ethernet addresses, 20 bytes - an IP header, and 46 bytes - a TCP header . There are two bits in the packet from the TCP header — syn and ack . In the first packet, syn = 1, ack = 0 . After that, the server sends us a packet of the same length (74 bytes), acknowledging receipt and allowing to connect, with syn = 1, ack = 1 . Then the client sends the third packet, but not 74, but 66 bytes in length. In the third packet, syn = 0, ack = 1 . This package finally establishes the connection and now we can receive and receive packets. It looks like a successful connection. If you would like to study TCP and other possible cases in more detail, you can read about it, for example, in the TANENTBAUM book “Computer Networks”.
>>> conn, addr = sock.accept()
We read information about the client. At this point, no packets are transmitted, we just take the data already recorded.
In the second terminal, enter
>>> protocol.send(sock, "Hello, localhost!")
With this line we send a message to the server using our mini-protocol.
At this moment we send a package with the following content to the server (excluding headers):
00:00:00:11:48:65:6c:6c:6f:2c:20:6c:6f:63:61:6c:68:6f:73:74:21
00: 00: 00: 11 - 0x11 , or 17 in decimal notation, is the length of the transmitted message (not the data, but the message, since the data, in this case, is 4 bytes of length + message).
48: 65: 6c: 6c: 6f: 2c: 20: 6c: 6f: 63: 61: 6c: 68: 6f: 73: 74: 21 - transmitted string Hello, localhost!
The server responds with another packet confirming receipt.
Finally, we read and decode the message in the first terminal, the client:
>>> protocol.recv(conn)
You can break the connection by typing in the second terminal
>>> sock.close()
The client sends the server a packet with the fin bit set, saying that it has no more data to send and wants to break the connection. In response, the server also sends a packet with the fin bit = 1 .
The same command in the first terminal will stop the server and it will stop listening.
Conclusion
After all we have done, we can easily transfer messages between client and server applications. The changes are not very different, so I did not mention the code listings in the article, but put the project on GitHub . In the next article I will talk about security. If you are interested in something else that you would like to hear about in the following articles, then write about it in the comments.
Source: https://habr.com/ru/post/269019/
All Articles