Mail Protocol Diagnostics
This article is about diagnosing mail protocols. It is intended for novice administrators who want to learn more about the tools for quickly testing the authorization / sending / receiving of e-mail messages both by the server and by the client. But it can also serve as a good reminder of the relevant teams for more experienced administrators.
The material is broken down as follows:
1. Introduction
2. Examples of sessions
3. Check authorization on the server (LOGIN, PLAIN, CRAM-MD5), Base64
4. Check SSL / TLS encryption
5. Analysis of mail traffic using tshark. SSL / TLS decryption
6. Links to materials
')
There are enough materials on individual points in the network, but everything is scattered in different places and, when there is a need to perform this or that operation, you have to recall authorization nuances, methods of fast encoding in base64, keys to openssl and tshark for various resources. Here everything is gathered together, and information about SSL / TLS traffic decryption has been added.
$ - an invitation in a regular shell, the command specified after it is executed from a regular user
# - invitation in the root shell, the command specified after it is executed with administrator rights
## - line with the comment
Client request in mail sessions is in bold.
The main ports used in the work of mail servers by RFC (documents regulating the work of the Internet and its main components):
Only the main ones are listed here; in addition to these, different server implementations can use other ports for their service purposes, for the user and administrative web interface, cluster nodes communication, etc.
The article uses telnet, openssl, tshark. For clarity, the interaction between the server and the client, the use of protocol commands. On a regular basis and to automate some processes, you can use utilities that hide all these details from us, but which are easier included in the scripts. Of these utilities, I can recommend the perl smtp-cli script (http://www.logix.cz/michal/devel/smtp-cli/), which has broad functionality, including SMTP authorization. I also recommend the imtest utility from cyrus-clients that you can use to test the IMAP protocol. smtp-sink , a postfix utility that emulates a mail server. With its help, you can debug the work of your mail client in the event that you do not have access to existing mail servers or the ability to enable detailed logging in the client settings.
With the help of nmap, you can quickly check if the ports are accessible from the outside, that is, if they listen to the programs and if the firewall is not closed:
This conclusion shows that the server has SMTP / IMAP ports available, but ports for
POP3 protocol.
Through netstat you can see not only the ports that are being listened to and used, as is often assumed, but also the processes associated with these ports. Here is the netstat output for the same mail server:
In this example, postfix is ​​used as the SMTP server and dovecot is used as the IMAP. POP3 is missing from the list, as this protocol is disabled in the dovecot settings as unused.
In modern distributions, the net-tools package is often not installed, it is considered obsolete. The ss utility from iproute is used as a replacement. It is more narrowly sharpened and in its own area is probably a more functional utility with the ability to configure filters as in tcpdump / tshark. But I, for one, do not like the way the output of information is formatted. To fix this a bit, you can use sed:
*) for ease of use, you can put the following bash function in ~ / .bashrc
Here are examples of sessions on SMTP / IMAP / POP3 protocols. The connection uses a telnet client, which is either installed by default in the system or installed from the repositories:
Commands entered in the text in bold.
Existing authorization methods: LOGIN, PLAIN, CRAM-MD5, DIGEST-MD5, GSSAPI, NTLM / MSN, EXTERNAL. Their list is even wider, we consider only the most common ones, namely LOGIN, PLAIN and CRAM-MD5.
The first step is to find out the list of methods supported by the server. For each of the mail protocols there are commands that allow you to get this data along with other information about the available protocol extensions. Please note that, depending on the settings of the mail server, LOGIN and PLAIN, which transmit data in the clear, may not be available without prior initialization of encryption via SSL / TLS
So, the output of the available authentication methods:
Some e-mail servers can display this information in the “server greeting,” for example, dovecot.
Where 'dGVzdA ==' is the login and 'dGVzdHBhc3M =' is the password in base64 format. About him a little lower. Please note that both login and password must be encoded without a line break.
Where 'dGVzdAB0ZXN0AHRlc3RwYXNz' is the login password in base64 format. Below will be considered options for converting to base64 format and back.
Unlike the previous CRAM-MD5 authorization methods, the password is not transmitted in the clear, the hash comparison is used instead. Manual verification of this method of authorization can be a problem, since it will be necessary to perform several transformations, and the time to enter commands is limited. To simplify the process, the following is a simple perl script that accepts a username, password and a “code word” (issued by the server) and converts them into a string in base64 format.
The script will need an additional perl module “Digest-HMAC”. In Debian / Ubuntu, you can find and install it as follows:
For RHEL / CentOS / Fedora:
In distributions whose repositories do not have this package (which is unlikely), you can use the installation module from CPAN.
Script and sample session using it:
Authorization involves the exchange of strings encoded in base64. For Linux, there are many utilities for converting to base64 and back. We will indicate a few, including how to launch them. For Windows, you can use cross-platform perl, python, php, examples will also be provided.
To decode, you need to add the first and last line. This can be done, for example, in the following ways;
or
The MMIME :: Base64 module comes bundled as standard.
SSL / TLS in two options is used to encrypt traffic in mail protocols between the client and the server. Using special ports, when connecting with which you first install SSL / TLS, after which normal mail traffic goes over it. This method, by the way, is deprecated, relative to SMTP for sure. The second option, the preferred one, is a connection to a normal port for the service and the session transition to an encrypted form using the STARTTLS extension.
To test the mail server over SSL / TLS, you can use the openssl utility, further acting as in a normal telnet session.
You can explicitly specify what to use for encryption, ssl3 or tls1, as well as specific algorithms:
View a list of supported protocols in your openssl version:
Below, in the tshark chapter, this feature will be used for practical purposes.
If you need more complex diagnostics in the case when the logs do not provide enough information about problems in the server or client, you can use tcpdump / wireshark to analyze the session itself between the client and the server. Both in real time and saving the dump session for later analysis. For quick analysis it is convenient to use the console version of wireshark - tshark. It requires root privileges.
Tshark provides information in a clear form and in use is quite simple.
Record traffic for further analysis using tcpdump | dumpcap utilities (from wireshark):
where HOSTIP is the IP address of the opposite party, server or client, the session with which we are analyzing. And the following reading:
In many cases, mail protocols actively use encryption, and in this way the session is no longer viewed. Nevertheless, this question as a whole is also resolved. tshark can decrypt SSL / TLS traffic "from the server side" if you have access to the server's private key (for the client there is an option using the Master-Key, learn more about wiki.wireshark.org/SSL ). Fortunately or unfortunately, wireshark with a private key can not decrypt all algorithms used. For example DHE- * EXP- *, EDH- * do not work. Perhaps some of these algorithms are added in later versions of the program.
In the process of testing, the openssl utility was used with an explicit indication when connecting with specific algorithms. Proven options with which traffic decryption was successful:
View a list of supported protocols in your openssl version:
To analyze a real session, you can turn off in the mail server configuration (only for the duration of the test!) All algorithms, except for obviously working ones.
Tshark runs on the server, where the key is, and the openssl client on the local computer. But, of course, it is not necessary, it is quite possible to run tshark on the client in another console, it just requires copying the private key to the local computer. And openssl can be launched in screen in a window next to tshark.
So run:
Here
Postfix example:
For ports where starttls are used instead of the port, it is recommended to use start_tls in the official documentation. For example,
To debug the SSL / TLS decryption process, use the
Example output of decrypted traffic:
Good luck in solving mail problems!
The material is broken down as follows:
1. Introduction
2. Examples of sessions
3. Check authorization on the server (LOGIN, PLAIN, CRAM-MD5), Base64
4. Check SSL / TLS encryption
5. Analysis of mail traffic using tshark. SSL / TLS decryption
6. Links to materials
')
1. Introduction
There are enough materials on individual points in the network, but everything is scattered in different places and, when there is a need to perform this or that operation, you have to recall authorization nuances, methods of fast encoding in base64, keys to openssl and tshark for various resources. Here everything is gathered together, and information about SSL / TLS traffic decryption has been added.
Legend
$ - an invitation in a regular shell, the command specified after it is executed from a regular user
# - invitation in the root shell, the command specified after it is executed with administrator rights
## - line with the comment
Client request in mail sessions is in bold.
Post ports
The main ports used in the work of mail servers by RFC (documents regulating the work of the Internet and its main components):
SMTP
- 25 / tcp SMTP (standard port)
- 465 / tcp SMTPS (outdated)
- 587 / tcp submission (port for customer service)
Pop3
- 110 / tcp POP3 (standard port)
- 995 / tcp POP3S (port with pre-installed SSL / TLS connection)
IMAP
- 143 / tcp IMAP (standard port)
- 993 / tcp IMAPS (port with pre-set SSL / TLS connection)
Only the main ones are listed here; in addition to these, different server implementations can use other ports for their service purposes, for the user and administrative web interface, cluster nodes communication, etc.
Used and recommended utilities
The article uses telnet, openssl, tshark. For clarity, the interaction between the server and the client, the use of protocol commands. On a regular basis and to automate some processes, you can use utilities that hide all these details from us, but which are easier included in the scripts. Of these utilities, I can recommend the perl smtp-cli script (http://www.logix.cz/michal/devel/smtp-cli/), which has broad functionality, including SMTP authorization. I also recommend the imtest utility from cyrus-clients that you can use to test the IMAP protocol. smtp-sink , a postfix utility that emulates a mail server. With its help, you can debug the work of your mail client in the event that you do not have access to existing mail servers or the ability to enable detailed logging in the client settings.
With the help of nmap, you can quickly check if the ports are accessible from the outside, that is, if they listen to the programs and if the firewall is not closed:
# nmap -v -p25,110,143,465,587,993,995 127.0.0.1 Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2014-10-31 15:59 MSK Initiating SYN Stealth Scan against localhost.localdomain (127.0.0.1) [7 ports] at 15:59 Discovered open port 25/tcp on 127.0.0.1 Discovered open port 465/tcp on 127.0.0.1 Discovered open port 143/tcp on 127.0.0.1 Discovered open port 993/tcp on 127.0.0.1 The SYN Stealth Scan took 0.00s to scan 7 total ports. Host localhost.localdomain (127.0.0.1) appears to be up ... good. Interesting ports on localhost.localdomain (127.0.0.1): PORT STATE SERVICE 25/tcp open smtp 110/tcp closed pop3 143/tcp open imap 465/tcp open smtps 587/tcp closed submission 993/tcp open imaps 995/tcp closed pop3s Nmap finished: 1 IP address (1 host up) scanned in 0.004 seconds Raw packets sent: 7 (308B) | Rcvd: 17 (724B) This conclusion shows that the server has SMTP / IMAP ports available, but ports for
POP3 protocol.
Through netstat you can see not only the ports that are being listened to and used, as is often assumed, but also the processes associated with these ports. Here is the netstat output for the same mail server:
# netstat -lnpvut ( -anpvut, ) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 477/dovecot tcp 0 0 0.0.0.0:2000 0.0.0.0:* LISTEN 477/dovecot tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 603/master tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 430/unbound tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 10042/sshd tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 603/master tcp 0 0 0.0.0.0:1025 0.0.0.0:* LISTEN 603/master tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 477/dovecot tcp 0 0 127.0.0.1:1953 0.0.0.0:* LISTEN 430/unbound tcp 0 0 127.0.0.1:1026 0.0.0.0:* LISTEN 603/master tcp 0 0 127.0.0.1:2025 0.0.0.0:* LISTEN 603/master tcp 0 0 :::22 :::* LISTEN 10042/sshd udp 0 0 127.0.0.1:53 0.0.0.0:* 430/unbound In this example, postfix is ​​used as the SMTP server and dovecot is used as the IMAP. POP3 is missing from the list, as this protocol is disabled in the dovecot settings as unused.
In modern distributions, the net-tools package is often not installed, it is considered obsolete. The ss utility from iproute is used as a replacement. It is more narrowly sharpened and in its own area is probably a more functional utility with the ability to configure filters as in tcpdump / tshark. But I, for one, do not like the way the output of information is formatted. To fix this a bit, you can use sed:
# ss -lntp | sed -r 's/\t/ /g' Recv-Q Send-Q Local Address:Port Peer Address:Port 0 0 *:143 *:* users:(("dovecot",477,6),("imap-login",14400,4),("imap-login",15370,4),("imap-login",15372,4)) 0 0 *:2000 *:* users:(("dovecot",477,8),("managesieve-log",10229,4),("managesieve-log",10230,4),("managesieve-log",21149,4)) 0 0 *:465 *:* users:(("master",603,31)) 0 0 127.0.0.1:53 *:* users:(("unbound",430,4)) 0 0 *:22 *:* users:(("sshd",10042,4)) 0 0 *:25 *:* users:(("master",603,19)) 0 0 *:1025 *:* users:(("master",603,12)) 0 0 *:993 *:* users:(("dovecot",477,7),("imap-login",14400,5),("imap-login",15370,5),("imap-login",15372,5)) 0 0 127.0.0.1:1953 *:* users:(("unbound",430,5)) 0 0 127.0.0.1:1026 *:* users:(("master",603,16)) 0 0 127.0.0.1:2025 *:* users:(("master",603,28)) 0 0 :::22 :::* users:(("sshd",10042,3)) *) for ease of use, you can put the following bash function in ~ / .bashrc
ss() { /sbin/ss $@ | sed -r 's/\t/ /g'; } 2. Examples of sessions
Here are examples of sessions on SMTP / IMAP / POP3 protocols. The connection uses a telnet client, which is either installed by default in the system or installed from the repositories:
Debian / Ubuntu
# apt-cache search telnet # apt-get install telnet RHEL / CentOS / Fedora
# yum search telnet # yum install telnet Commands entered in the text in bold.
SMTP
$ telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. 220 mailserver at mail.server.net greets you. Make love not war! HELO localhost.localdomain 250 mail.server.net MAIL FROM:<> 250 2.1.0 Ok RCPT TO:<user@mail.server.net> 250 2.1.5 Ok DATA 354 End data with <CR><LF>.<CR><LF> FROM: root@localhost.localdomain TO: user@mail.server.net SUBJECT: test mail from test subject test body . 250 2.0.0 Ok: queued as 1CF5FC0AAE QUIT 221 2.0.0 Bye Connection closed by foreign host. IMAP
$ telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. * OK IMAP Server at mail.server.net ready 001 LOGIN user@mail.server.net testpass 001 OK completed 002 CAPABILITY * CAPABILITY IMAP4 IMAP4REV1 ACL NAMESPACE UIDPLUS IDLE LITERAL+ QUOTA ID MULTIAPPEND LISTEXT CHILDREN BINARY LOGIN-REFERRALS STARTTLS AUTH=LOGIN AUTH=PLAIN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 AUTH=MSN 002 OK completed 003 SELECT Inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent)] limited * 7214 EXISTS * 0 RECENT * OK [UIDVALIDITY 306349424] UIDs valid * OK [UNSEEN 1] message 1 is first unseen 003 OK [READ-WRITE] SELECT completed 004 FETCH 7214 body[header] * 7214 FETCH (BODY[header] {639} Return-Path: <> X-Antispam-passed: yes X-Antispam: yes X-Real-To: user@mail.server.net Received: from [127.0.0.1] (HELO mail.server.net) by mail.server.net ( SMTP 4.1.8) with ESMTP id 22561074 for user@mail.server.net; Sat, 01 Nov 2014 03:21:16 +0300 Received: from localhost.localdomain (localhost [127.0.0.1]) by mail.server.net (Postfix) with SMTP id 1CF5FC0AAE for <user@mail.server.net>; Sat, 1 Nov 2014 03:20:09 +0300 (MSK) FROM: root@localhost.localdomain TO: user@mail.server.net SUBJECT: test mail from test subject Message-Id: <20141101002009.1CF5FC0AAE@mail.server.net> Date: Sat, 1 Nov 2014 03:20:09 +0300 (MSK) FLAGS (\Seen)) 004 OK completed 004 FETCH 7214 body * 7214 FETCH (BODY ("text" "plain" NIL NIL NIL "8bit" 13 2)) 004 OK completed 004 FETCH 7214 body[] * 7214 FETCH (BODY[] {652} Return-Path: <> X-Antispam-passed: yes X-Antispam: yes X-Real-To: user@mail.server.net Received: from [127.0.0.1] (HELO mail.server.net) by mail.server.net ( SMTP 4.1.8) with ESMTP id 22561074 for user@mail.server.net; Sat, 01 Nov 2014 03:21:16 +0300 Received: from localhost.localdomain (localhost [127.0.0.1]) by mail.server.net (Postfix) with SMTP id 1CF5FC0AAE for <user@mail.server.net>; Sat, 1 Nov 2014 03:20:09 +0300 (MSK) FROM: root@localhost.localdomain TO: user@mail.server.net SUBJECT: test mail from test subject Message-Id: <20141101002009.1CF5FC0AAE@mail.server.net> Date: Sat, 1 Nov 2014 03:20:09 +0300 (MSK) test body ) 004 OK completed 005 LOGOUT * BYE IMAP closing connection 005 OK completed Connection closed by foreign host. Pop3
$ telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. +OK POP3 Server 4.1.8 ready <137.1414802293@mail.server.net> USER test@mail.server.net +OK please send the PASS PASS testpass +OK 7214 messages (174404489 bytes) NOOP +OK cool TOP 7214 +OK message follows Return-Path: <> X-Antispam-passed: yes X-Antispam: yes X-Real-To: test@mail.server.net Received: from [127.0.0.1] (HELO mail.server.net) by mail.server.net ( SMTP 4.1.8) with ESMTP id 22561074 for test@mail.server.net; Sat, 01 Nov 2014 03:21:16 +0300 Received: from localhost.localdomain (localhost [127.0.0.1]) by mail.server.net (Postfix) with SMTP id 1CF5FC0AAE for <test@mail.server.net>; Sat, 1 Nov 2014 03:20:09 +0300 (MSK) FROM: root@localhost.localdomain TO: test@mail.server.net SUBJECT: test mail from test subject Message-Id: <20141101002009.1CF5FC0AAE@mail.server.net> Date: Sat, 1 Nov 2014 03:20:09 +0300 (MSK) . RETR 7214 +OK 652 bytes will follow Return-Path: <> X-Antispam-passed: yes X-Antispam: yes X-Real-To: test@mail.server.net Received: from [127.0.0.1] (HELO mail.server.net) by mail.server.net ( SMTP 4.1.8) with ESMTP id 22561074 for test@mail.server.net; Sat, 01 Nov 2014 03:21:16 +0300 Received: from localhost.localdomain (localhost [127.0.0.1]) by mail.server.net (Postfix) with SMTP id 1CF5FC0AAE for <test@mail.server.net>; Sat, 1 Nov 2014 03:20:09 +0300 (MSK) FROM: root@localhost.localdomain TO: test@mail.server.net SUBJECT: test mail from test subject Message-Id: <20141101002009.1CF5FC0AAE@mail.server.net> Date: Sat, 1 Nov 2014 03:20:09 +0300 (MSK) test body . DELE 7214 +OK marked deleted QUIT +OK POP3 Server connection closed Connection closed by foreign host. 3. Check authorization on the server
Existing authorization methods: LOGIN, PLAIN, CRAM-MD5, DIGEST-MD5, GSSAPI, NTLM / MSN, EXTERNAL. Their list is even wider, we consider only the most common ones, namely LOGIN, PLAIN and CRAM-MD5.
The first step is to find out the list of methods supported by the server. For each of the mail protocols there are commands that allow you to get this data along with other information about the available protocol extensions. Please note that, depending on the settings of the mail server, LOGIN and PLAIN, which transmit data in the clear, may not be available without prior initialization of encryption via SSL / TLS
So, the output of the available authentication methods:
SMTP protocol
EHLO domainname command
$ telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. 220 mailserver ESMTP ready. EHLO localhost.localdomain 250-mal.server.net 250-PIPELINING 250-SIZE 104857600 250-ETRN 250-STARTTLS 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5 250-ENHANCEDSTATUSCODES 250 8BITMIME ^] telnet> quit Connection closed. IMAP protocol
Team 001 CAPABILITY
Some e-mail servers can display this information in the “server greeting,” for example, dovecot.
$ telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5] Dovecot ready. 001 CAPABILITY * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS XEXEC QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5 001 OK Capability completed. 002 LOGOUT * BYE Logging out 002 OK Logout completed. Connection closed by foreign host. POP3 protocol
AUTH or CAPA Commands
$ telnet pop.mail.ru 110 Trying 217.69.139.74... Connected to pop.mail.ru. Escape character is '^]'. +OK AUTH +OK methods supported: LOGIN PLAIN . CAPA +OK Capability list follows TOP USER LOGIN-DELAY 120 EXPIRE NEVER UIDL IMPLEMENTATION Mail.Ru SASL LOGIN PLAIN STLS . QUIT +OK POP3 server at signing off Connection closed by foreign host. Authorization Examples and Format Used
LOGIN
SMTP protocol
$ telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. 220 mail.server.net ESMTP Server EHLO client.server.net 250-mail.server.net Hello client.server.net 250-AUTH LOGIN PLAIN CRAM-MD5 DIGEST-MD5 GSSAPI 250-ENHANCEDSTATUSCODES 250 STARTTLS AUTH LOGIN 334 VXNlcm5hbWU6 dGVzdA== 334 UGFzc3dvcmQ6 dGVzdHBhc3M= 235 2.7.0 Authentication successful QUIT 221 2.0.0 Bye Where 'dGVzdA ==' is the login and 'dGVzdHBhc3M =' is the password in base64 format. About him a little lower. Please note that both login and password must be encoded without a line break.
Plain
SMTP protocol
$ telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. 220 mail.server.net ESMTP Server EHLO client.server.net 250-mail.server.net Hello client.server.net 250-AUTH LOGIN PLAIN CRAM-MD5 DIGEST-MD5 GSSAPI 250-ENHANCEDSTATUSCODES 250 STARTTLS AUTH PLAIN dGVzdAB0ZXN0AHRlc3RwYXNz 235 2.7.0 Authentication successful QUIT 221 2.0.0 Bye Where 'dGVzdAB0ZXN0AHRlc3RwYXNz' is the login password in base64 format. Below will be considered options for converting to base64 format and back.
CRAM-MD5
Unlike the previous CRAM-MD5 authorization methods, the password is not transmitted in the clear, the hash comparison is used instead. Manual verification of this method of authorization can be a problem, since it will be necessary to perform several transformations, and the time to enter commands is limited. To simplify the process, the following is a simple perl script that accepts a username, password and a “code word” (issued by the server) and converts them into a string in base64 format.
The script will need an additional perl module “Digest-HMAC”. In Debian / Ubuntu, you can find and install it as follows:
# apt-cache search perl | grep -i digest # apt-get install libdigest-hmac-perl For RHEL / CentOS / Fedora:
# yum search perl | grep -i digest # yum install perl-Digest-HMAC In distributions whose repositories do not have this package (which is unlikely), you can use the installation module from CPAN.
Script and sample session using it:
#!/usr/bin/perl -W use strict; use MIME::Base64 qw(encode_base64 decode_base64); use Digest::HMAC_MD5; die "Usage: $0 username password ticket\n" unless $#ARGV == 2; my ($username, $password, $ticket64) = @ARGV; my $ticket = decode_base64($ticket64) or die ("Unable to decode Base64 encoded string '$ticket64'\n"); my $password_md5 = Digest::HMAC_MD5::hmac_md5_hex($ticket, $password); print encode_base64 ("$username $password_md5", ""); SMTP protocol
$ telnet 127.0.0.1 25 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. 220 mail.server.net ESMTP Server EHLO client.server.net 250-mail.server.net Hello client.server.net 250-AUTH LOGIN PLAIN CRAM-MD5 DIGEST-MD5 GSSAPI 250-ENHANCEDSTATUSCODES 250 STARTTLS AUTH CRAM-MD5 ## , : PDMzMjE2NDkzMTA1OTExNDQuMTQxNDc5NTExOUBtYWlsLnNlcnZlci5uZXQ+ dGVzdCAxNTU0YTQwNzA1NTgxZjUwZmI1MmNjZDhlZDhjM2EyYg== 235 2.7.0 Authentication successful QUIT 221 2.0.0 Bye # ./md5cram.pl test testpass PDMzMjE2NDkzMTA1OTExNDQuMTQxNDc5NTExOUBtYWlsLnNlcnZlci5uZXQ+ dGVzdCAxNTU0YTQwNzA1NTgxZjUwZmI1MmNjZDhlZDhjM2EyYg== IMAP protocol
$ telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5] Dovecot ready. 01 AUTHENTICATE CRAM-MD5 + PDgxOTAyMjA2NTYwNzcyMzEuMTQxNDc5NzA3MkBtYWlsLnNlcnZlci5uZXQ+ dGVzdCA1YTZlNjYwMDlmZGJlZWNjYWRlNDY5M2FlMjU5YTA2ZQ== 01 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS XEXEC QUOTA] Logged in 02 LOGOUT * BYE Logging out 02 OK Logout completed. Connection closed by foreign host. # ./md5cram.pl test testpass PDgxOTAyMjA2NTYwNzcyMzEuMTQxNDc5NzA3MkBtYWlsLnNlcnZlci5uZXQ+ dGVzdCA1YTZlNjYwMDlmZGJlZWNjYWRlNDY5M2FlMjU5YTA2ZQ== Converting text to and from base64
Authorization involves the exchange of strings encoded in base64. For Linux, there are many utilities for converting to base64 and back. We will indicate a few, including how to launch them. For Windows, you can use cross-platform perl, python, php, examples will also be provided.
Utility (package)
base64 (coreutils)
$ printf 'test\0test\0testpass' | base64 dGVzdAB0ZXN0AHRlc3RwYXNz $ echo dGVzdAB0ZXN0AHRlc3RwYXNz | base64 -d testtesttestpass uueencode / uudecode (sharutils)
$ printf 'test\0test\0testpass' | uuencode -m - begin-base64 644 - dGVzdAB0ZXN0AHRlc3RwYXNz ==== To decode, you need to add the first and last line. This can be done, for example, in the following ways;
printf 'begin-base64 644 -\ndGVzdAB0ZXN0AHRlc3RwYXNz\n====' | uudecode or
$ uudecode<<EOF begin-base64 644 - dGVzdAB0ZXN0AHRlc3RwYXNz ==== EOF mmencode (xemacs21-bin)
$ printf 'test\0test\0testpass' | mmencode dGVzdAB0ZXN0AHRlc3RwYXNz $ echo dGVzdAB0ZXN0AHRlc3RwYXNz | mmencode -u testtesttestpass python (python)
$ printf 'test\0test\0testpass' | python -m base64 dGVzdAB0ZXN0AHRlc3RwYXNz $ echo dGVzdAB0ZXN0AHRlc3RwYXNz | python -m base64 -d php (php-cli)
$ printf 'test\0test\0testpass' | php -r 'echo base64_encode(fgets(STDIN));' dGVzdAB0ZXN0AHRlc3RwYXNz $ php -r 'echo base64_decode($argv[1]);' dGVzdAB0ZXN0AHRlc3RwYXNz testtesttestpass perl (perl)
The MMIME :: Base64 module comes bundled as standard.
$ perl -MMIME::Base64 -e 'print encode_base64("test\0test\0testpass")' dGVzdAB0ZXN0AHRlc3RwYXNz $ perl -MMIME::Base64 -e 'print decode_base64("dGVzdAB0ZXN0AHRlc3RwYXNz")' testtesttestpass openssl (openssl)
$ printf 'test\0test\0testpass' | openssl base64 dGVzdAB0ZXN0AHRlc3RwYXNz $ echo dGVzdAB0ZXN0AHRlc3RwYXNz | openssl base64 -d testtesttestpass 4. Check SSL / TLS encryption.
SSL / TLS in two options is used to encrypt traffic in mail protocols between the client and the server. Using special ports, when connecting with which you first install SSL / TLS, after which normal mail traffic goes over it. This method, by the way, is deprecated, relative to SMTP for sure. The second option, the preferred one, is a connection to a normal port for the service and the session transition to an encrypted form using the STARTTLS extension.
To test the mail server over SSL / TLS, you can use the openssl utility, further acting as in a normal telnet session.
SMTP
$ openssl s_client -starttls smtp -crlf -connect mail.truevds.ru:25 $ openssl s_client -starttls smtp -crlf -connect mail.truevds.ru:587 $ openssl s_client -crlf -connect mail.truevds.ru:465 Pop3
$ openssl s_client -connect mail.truevds.ru:995 $ openssl s_client -starttls pop3 -crlf -connect mail.truevds.ru:110 IMAP
$ openssl s_client -crlf -connect mail.truevds.ru:993 $ openssl s_client -starttls imap -crlf -connect mail.truevds.ru:143 You can explicitly specify what to use for encryption, ssl3 or tls1, as well as specific algorithms:
$ openssl s_client -ssl3 -starttls smtp -crlf -connect mail.truevds.ru:25 View a list of supported protocols in your openssl version:
$ openssl ciphers -ssl3 $ openssl ciphers -tls1 Below, in the tshark chapter, this feature will be used for practical purposes.
5. Analysis of mail traffic using tshark. SSL / TLS decryption
If you need more complex diagnostics in the case when the logs do not provide enough information about problems in the server or client, you can use tcpdump / wireshark to analyze the session itself between the client and the server. Both in real time and saving the dump session for later analysis. For quick analysis it is convenient to use the console version of wireshark - tshark. It requires root privileges.
Tshark provides information in a clear form and in use is quite simple.
SMTP
# tshark -i eth0 -f "port 25" -R smtp IMAP
# tshark -i eth0 -f "port 143" -R imap Pop3
# tshark -i eth0 -f "port 110" -R pop Record traffic for further analysis using tcpdump | dumpcap utilities (from wireshark):
# tcpdump -s0 -nn -i eth0 -w smtps.pcap port 465 and host HOSTIP # dumpcap -s0 -i eth0 -w smtp.pcap -f 'port 25 and host HOSTIP' where HOSTIP is the IP address of the opposite party, server or client, the session with which we are analyzing. And the following reading:
# tshark -n -r smtp.pcap -R smtp In many cases, mail protocols actively use encryption, and in this way the session is no longer viewed. Nevertheless, this question as a whole is also resolved. tshark can decrypt SSL / TLS traffic "from the server side" if you have access to the server's private key (for the client there is an option using the Master-Key, learn more about wiki.wireshark.org/SSL ). Fortunately or unfortunately, wireshark with a private key can not decrypt all algorithms used. For example DHE- * EXP- *, EDH- * do not work. Perhaps some of these algorithms are added in later versions of the program.
In the process of testing, the openssl utility was used with an explicit indication when connecting with specific algorithms. Proven options with which traffic decryption was successful:
- ssl3: RC4-SHA, RC4-MD5, DES-CBC-SHA, AES128-SHA
- tls1: RC4-MD5, AES256-SHA, DES-CBC-SHA, DES-CBC3-SHA
View a list of supported protocols in your openssl version:
# openssl ciphers -ssl3 # openssl ciphers -tls1 To analyze a real session, you can turn off in the mail server configuration (only for the duration of the test!) All algorithms, except for obviously working ones.
Tshark runs on the server, where the key is, and the openssl client on the local computer. But, of course, it is not necessary, it is quite possible to run tshark on the client in another console, it just requires copying the private key to the local computer. And openssl can be launched in screen in a window next to tshark.
So run:
# tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,25,smtp,/etc/pki/tls/private/server.key" -R smtp $ printf "EHLO RC4-MD5\nEXIT" | openssl s_client -starttls smtp -crlf -tls1 -cipher RC4-MD5 -connect mail.truevds.ru:25 # tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,465,smtp,/etc/pki/tls/private/server.key" -R smtp $ printf "EHLO RC4-MD5\nEXIT" | openssl s_client -ssl3 -cipher RC4-SHA -connect mail.truevds.ru:465 # tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,143,imap,/etc/pki/tls/private/server.key" -R imap $ printf "* CAPABILITY\nLOGOUT" | openssl s_client -starttls imap -crlf -tls1 -cipher RC4-MD5 -connect mail.truevds.ru:143 # tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,993,imap,/etc/pki/tls/private/server.key" -R imap $ printf "* CAPABILITY\nLOGOUT" | openssl s_client -crlf -ssl3 -cipher RC4-MD5 -connect mail.truevds.ru:993 # tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,110,pop,/etc/pki/tls/private/server.key" -R pop $ printf "USER RC4-MD5\nEXIT" | openssl s_client -starttls pop -crlf -tls1 -cipher RC4-MD5 -connect mail.truevds.ru:110 # tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,995,pop,/etc/pki/tls/private/server.key" -R pop $ printf "USER RC4-MD5\nEXIT" | openssl s_client -crlf -ssl3 -cipher RC4-MD5 -connect mail.truevds.ru:995 Here
94.127.66.53 is the ip address of the server to which the client connects, /etc/pki/tls/private/server.key is the path to the private key of the server. The private key is usually located in /etc/pki or /etc/ssl , depending on the server. This information can be viewed in the settings of the mail server itself.Postfix example:
$ grep key_file /etc/postfix/main.cf smtpd_tls_key_file = /etc/pki/tls/private/server.key smtp_tls_key_file = /etc/pki/tls/private/server.key For ports where starttls are used instead of the port, it is recommended to use start_tls in the official documentation. For example,
ssl.keys_list:94.127.66.53,start_tls,smtp,/etc/pki/tls/private/server.key instead of ssl.keys_list:94.127.66.53,25,smtp,/etc/pki/tls/private/server.key . But this option did not work for me, traffic only showed up before encryption was initialized.To debug the SSL / TLS decryption process, use the
-o "ssl.debug_file: /tmp/debug.log"Example output of decrypted traffic:
# tshark -i eth0 -n -o "ssl.keys_list:94.127.66.53,25,smtp,/etc/pki/tls/private/server.key" -R "smtp" Running as user "root" and group "root". This could be dangerous. Capturing on eth0 0.178964 94.127.66.21 -> 94.127.66.53 SMTP C: EHLO RC4-MD5 | EXIT 0.179357 94.127.66.53 -> 94.127.66.21 SMTP 250-mail.truevds.ru | 250-PIPELINING | 250-SIZE 104857600 | 250-ETRN | 6. Links to materials
- Wireshark Network Analyzer Documentation: wiki.wireshark.org
- SMTP: RFC 2821
- SMTP Authorization: RFC 4954
- POP3: RFC 3501
- IMAP: RFC 1939
- Smtp-cli website: www.logix.cz/michal/devel/smtp-cli
Good luck in solving mail problems!
Source: https://habr.com/ru/post/262819/
All Articles