Recently, when developing a distributed traffic analyzer, I had the task of designing an authentication system between the client and the server. Moreover, it was necessary to design the system for two different situations:
- when the client and server communicate in a trusted (local) network;
- The client and server communicate through a global, unsecured network.
The differences lie in the fact that when a client and a server interact through a local network, when designing a system, threats of intrusion from outside were excluded (the threat of a hacker sitting next to you, who redirected your traffic to your computer, was minimized). In addition, the interaction between the client and server in the local network means that the client and server have no access to the Internet or are not desired, that is, there is also no access to certificate authorities, and, as a result, it is impossible to fully use authentication using public keys.
Unlike a local network, where an attacker somehow needs to connect to it (and if good system administrators manage such a network, this is quite problematic), when a client and server interact through an unprotected network, you cannot be sure that no one listens for traffic. Therefore, an authentication algorithm is required here that is different from the algorithm for interaction in the local network.
')
In addition, some additional restrictions were imposed on the authentication system.
In general, the limitations can be described as follows:
- strong encryption of traffic between the client and the server is required;
- The client should work immediately after installation on the computer, without requesting a password from the user. That is, the employee does not need to know at all about the existence of such a client installed on his computer. All settings for the client come from the server, and the client, in turn, should simply be able to verify his identity in order to send encrypted traffic;
- if curious employees have access to a client and its private key, then the maximum that they can do is to decrypt messages only for a certain period of time, for which this key was created. That is, forward secrecy (security of future messages) and backward secrecy (secrecy of past messages) are partially provided;
- if employees start to install clients on local computers without need, it will not be critical, since the client does not have the ability to disrupt the server, or to gain access to encrypted data. In addition, the server can ban such clients.
Based on these limitations, the following protocols and algorithms were chosen:
- Authentication protocol based on a shared secret key using HMAC is used for authentication in the local network, and Diffie-Hellman's algorithm is used to generate the shared secret key (I will not write about the algorithm itself here, since there are already many detailed articles on the Habré about him).
- for authentication between the client and the server in an unsecured network, it is supposed to use an authentication protocol using public key encryption using the RSA algorithm (you can see about RSA on Wikipedia).
Shared Secret Authentication Using HMAC
This type of authentication implies that the client and the server have a certain secret key that only two of them know. We generated such a key using the Diffie-Hellman algorithm.
HMAC (short for English hash-based message authentication code, message identification hash code). That is, it is a mechanism that uses cryptographic hash functions in combination with a secret key.
The protocol authentication sequence is as follows:

N, N - 128 bit number, randomly selected by the client and server separately;
K, C - identifiers of the client and server (for example, IP address);
HMAC - hash message identification code;
Ks is a shared secret key.
In the first step, the client sends a randomly selected number to the server so that it can encrypt it with a key common to them, and thereby confirm that it is in fact a server and not an attacker who pretends to be it (although as we will see later, in our example this number is not encrypted, but is converted with other data into a hash, which does not change the essence of the process).
In the second step, the server sends the client a random number Nc, as well as an HMAC hash, formed by hashing the values ​​Nk, Nc, K, C, and Ks using the HMAC function. Because the client knows NC, NC, K, C, and Ks, then he can generate his HMAC hash, and compare it with what came from the server. If the hashes match, then the server is what it claims to be. At the same time, the attacker could not form a HMAC hash (NC, NC, K, C, Ks), because at least he should not know the secret key Ks.
In the third step, it was the client's turn to confirm his “identity”. The process is carried out by analogy with the second step.
Authentication using public key encryption
This type of authentication is used when both the client and the server have certificates. The sequence of steps of this type of authentication is shown below.

N, N - 128 bit number, randomly selected by the client and server separately;
K, C - identifiers of the client and server (for example, IP address);
Ks - shared secret key;
Ec, Ec are the public keys of the server and the client, respectively, which are transmitted in the form of certificates.
First, the client requests the public key of the server (EU) from the certification authority. Then he encrypts with his help a randomly selected number and his identifier (step 3). Only a real server can decrypt this message. Therefore, the server, receiving this message, decrypts it, and encrypts the client’s public key (which it receives from the certification authority) with the client’s random number, its random number, and the shared secret key (step 6). Further, if the client is the one who claims to be, decrypts the server message, finds out the shared secret key, and to confirm that he was able to decrypt the server message, he randomly sends the number Nc to the server by encrypting it with the shared secret key (step 7).
When designing an authentication system, in contrast to the standard protocol, it is assumed that clients will not have their certificate, but will send their public key to the server, encrypting it with the server's previously public key obtained using the certificate. This will avoid the need to obtain multiple certificates for each client, which would entail the need for the user of this client to be distracted by its configuration, which is contrary to the limitations of the system.

Additional protection is provided by changing the shared secret key (Ks) at random intervals (Tk). Read more about this in the section “Analysis of the system for protection from attacks”.
Authentication System Algorithm
Further, all the stages of establishing a connection have been described using flowcharts.
The client authentication system algorithm is as follows:

From the server side:

After successful authentication, encrypted traffic can be transmitted. It is assumed that traffic wakes up encrypted using the AES algorithm, using the secret key Ks. Why AES? Because it is the best symmetric encryption algorithm to date. More information about him can be found on the Internet.
System analysis for protection from attacks
Next, I simulated various situations that may occur, and analyzed the resilience of the system to them.
Diffie-Hellman algorithm vulnerability to MITM attacks
This is perhaps the main drawback of this algorithm. If we assume that the attacker impersonates the server to the client and the client to the server (for example, using ARP-spoofing), then all traffic will be routed through the attacker.

One can only protect oneself from such attacks by properly thinking through the network architecture and the rules of the information security company. For example, you can install software to detect ARP-spoofinga and similar attacks.
Connecting clients to the server without the need
What can happen if someone gets a client of our analyzer?
On the one hand, because the server does not transmit any important information to the client (only commands), and the client, in turn, cannot get access to important server data and only sends traffic, this will not cause any serious inconvenience.

The only thing that can cause inconvenience is that the server database may increase indecently, but it is the server’s and administrator’s concern to disconnect unnecessary clients from the network (as it takes a long time to fill the database).
On the other hand, the client can initiate a DOS attack by sending multiple requests to establish a shared key, then deleting the received key and sending a new request to obtain the secret key.

Here you can simply ban on the time of such customers. Also, the client cannot initialize the update of the secret key. The only one who knows the duration of the shared secret key is the server itself. And only he can send a request to update the shared secret key.
The attacker gets the client's secret key and traffic from this client
In this situation, it is assumed that the attacker has acquired the secret key and client traffic.

m - network packet (message) that is encrypted with a secret key.
As mentioned earlier, the establishment of a shared secret key between the agent and the server in the trusted zone is performed using the Diffie-Hellman algorithm. Unlike the standard Diffie-Hellman, in my case, it is assumed that the shared key (Ks) is changed at a random time interval that lies in the time interval (Tmin, Tmax), where Tmin and Tmax are set in the server-side settings of the application.
This method allows to partially provide forward secrecy (secrecy of future messages) and backward secrecy (secrecy of past messages). That is, if we assume that the attacker inside the enterprise has the previously recorded traffic, and he managed to get the current shared secret key Ks (i), he can only decrypt such traffic that was transmitted during the interval (Tmin (i), Tmax ( i)), where i is the current number of the generated shared secret key.
Conclusion
This article has described the design of an authentication system based on widely known authentication protocols. A brief description of a shared secret key authentication protocol using HMAC and an authentication protocol using public key encryption was provided.
At the end, a brief analysis was made of the susceptibility of the designed system to computer attacks.
PS I would welcome comments, corrections and reasoned criticism.