Mozilla Thunderbird OAuth authorization: from inception to release

Some time ago we talked about how Mail.Ru implemented mail collection using the OAuth 2.0 protocol . We continue to improve mail security and push the OAuth 2.0 standard to the masses. And today we will tell about how we added OAuth authentication to the Mozilla Thunderbird email client. In this example, we will analyze the process of introducing a new feature into an open source product, from creating a ticket to a release. If you have long wanted to make your first pull request, but did not know how, - read our story.

1. The general scheme of work

On user actions Thunderbird opens a web view with an address for OAuth authorization. If the user has successfully passed the authorization procedure and agreed to provide the application with access to his data, then we redirect the user to the address specified in the redirect_uri parameter. So the application will receive an authorization token and can use it to work with our postal service. Address to request a token:

 https://o2.mail.ru/login ?response_type=code &client_id=< > &redirect_uri=http%3A%2F%2Flocalhost

It is worth noting that the application sets the localhost value for the redirect_uri parameter itself, and does not transmit the state parameter (used by the client to support the connection between the request and the callback). Below is a diagram of the interaction of the application with the service:

2. How is the integration process, the main stages

Although the integration process itself is fairly simple and does not take much time, it is still worth talking about specific points that should be planned in advance.

Since Thunderbird is a Mozilla product, we immediately went to MDN . So we quickly got a general idea of the main stages of integration:

Ticket to add mail client.
Ticket to add configuration to the ISP-base.
Patch to the comm-central repository.
Patch to ISP database.
Test build.
Preserving backward compatibility.
Testing functionality in early releases.
Release Testing

Next, we consider each stage separately.

2.1. Ticket to add mail client

Before you put a ticket, make sure that no one has done this before you. Setting a ticket is a key step in solving any similar problem, so it is very important to fill in all the required fields correctly:

Product : The comm-central repository structure is divided into independent products. We had no problems with this, since the name of the product MailNews is immediately mentioned on the start page .
Component : Opposite this item there is a hint, however, here and so intuitively it is clear that the work is connected with the network, therefore we choose Networking.
Version : To understand which release version to choose, you should refer to the release list page. However, this information will not be enough, since it is important for us to understand at what stage the release is not yet released. With this we will help release calendar . More information about the release cycle can be found on the corresponding page . But if you still doubt which release version to choose, feel free to ask your question on the mailing list or the IRC channel . If absolutely necessary, ticket reviewers will help you.
Platform : In our case, the product is platform- all ( all ).
Importance : As we expand the functionality, the type will be enhancement .
Keywords : Keyword list is limited. A quick search for similar tickets prompted to choose a feature , user-doc-needed .

Advice: in order to not miss any stage, add a calendar to yourself.

2.2. Ticket to add configuration to the ISP-base

We got a good reviewer who helped with filling out most of the fields and release. See an example of our ticket.

2.3. Patch in the comm-central repository

If you noticed, the Mozilla community is very sensitive about documenting their products. Guidance on the assembly of the product, the style of writing code, etc. All references to this are located in one place and do not require, as is often the case with other products, the passage of a certain quest. I’ll say right away that there is no “rocket science” in adding a new OAuth provider to Thunderbird, this becomes clear after the repository grep and a quick acquaintance with the source code. Although there were quite a few files with the OAuth keyword:

 ➜ hg clone http://hg.mozilla.org/comm-central ... ➜ hg grep --ignore-case --files-with-matches oauth | wc -l 91

Intuition suggested that everything should be easier. And we were not mistaken when we opened the first file from the list:

 mailnews/base/util/OAuth2Providers.jsm

I will not torment, just look diff:

 ➜ hg log -p -l 1

 changeset: [draft] 18512:a2f404184ac0 support_oauth_mail_ru_1231642 tip author: Alexander Abashkin <monolithed@gmail.com> date: Mon, 14 Dec 2015 15:17:09 +0300 (4 months ago) summary: Bug 1231642 - Log in to Mail.Ru (IMAP/SMTP) using OAuth M mailnews/base/util/OAuth2Providers.jsm diff --git a/mailnews/base/util/OAuth2Providers.jsm b/mailnews/base/util/OAuth2Providers.jsm --- a/mailnews/base/util/OAuth2Providers.jsm +++ b/mailnews/base/util/OAuth2Providers.jsm @@ -10,32 +10,41 @@ var EXPORTED_SYMBOLS = ["OAuth2Providers var {classes: Cc, interfaces: Ci, results: Cr, utils: Cu} = Components; // map of hostnames to [issuer, scope] var kHostnames = new Map([ ["imap.googlemail.com", ["accounts.google.com", "https://mail.google.com/"]], ["smtp.googlemail.com", ["accounts.google.com", "https://mail.google.com/"]], ["imap.gmail.com", ["accounts.google.com", "https://mail.google.com/"]], ["smtp.gmail.com", ["accounts.google.com", "https://mail.google.com/"]], + ["imap.mail.ru", ["o2.mail.ru", "mail.imap"]], + ["smtp.mail.ru", ["o2.mail.ru", "mail.imap"]], ]); // map of issuers to appKey, appSecret, authURI, tokenURI var kIssuers = new Map ([ ["accounts.google.com", [ '406964657835-aq8lmia8j95dhl1a2bvharmfk3t1hgqj.apps.googleusercontent.com', 'xxxxxxxxxxxx', 'https://accounts.google.com/o/oauth2/auth', 'https://www.googleapis.com/oauth2/v3/token' ]], + ["o2.mail.ru", [ + 'thunderbird', + 'xxxxxxxxxxxx', + 'https://o2.mail.ru/login', + 'https://o2.mail.ru/token' + ]], ]);

If you paid attention, then, unfortunately, we have not yet supported the new protocol , which allows you to dynamically register a client, but we are working on it! And then the patch attach by ticket number:

 hg diff -g > 1231642.patch

PS Be prepared for the fact that cloning the repository requires up to 5 GB of free disk space.

2.4. Patch to ISP database

This configuration is necessary to select the protocol to be used by default. An example of autoconfig file: https://autoconfig.thunderbird.net/v1.1/mail.ru . The ISP SVN repository is located at the following address:

 https://svn.mozilla.org/mozillamessaging.com/sites/autoconfig.mozillamessaging.com/

Since we have already dealt with this config earlier, it will be easier to show diff than telling:

 ➜ svn diff trunk/mail.ru | vi -R -

 --- trunk/mail.ru| (revision 150325) +++ trunk/mail.ru| (working copy) @@ -13,6 +13,7 @@ <incomingServer type="imap"> <hostname>imap.mail.ru</hostname> <port>993</port> <socketType>SSL</socketType> <username>%EMAILADDRESS%</username> + <authentication>OAuth2</authentication> <authentication>password-cleartext</authentication> </incomingServer> <incomingServer type="imap"> <hostname>imap.mail.ru</hostname> <port>143</port> <socketType>STARTTLS</socketType> <username>%EMAILADDRESS%</username> + <authentication>OAuth2</authentication> <authentication>password-cleartext</authentication> </incomingServer> <outgoingServer type="smtp"> <hostname>smtp.mail.ru</hostname> <port>465</port> <socketType>SSL</socketType> <username>%EMAILADDRESS%</username> + <authentication>OAuth2</authentication> <authentication>password-cleartext</authentication> </outgoingServer> <outgoingServer type="smtp"> <hostname>smtp.mail.ru</hostname> <port>587</port> <socketType>STARTTLS</socketType> <username>%EMAILADDRESS%</username> + <authentication>OAuth2</authentication> <authentication>password-cleartext</authentication> </outgoingServer>

At this stage, you should not hurry, even if your server already supports OAuth authorization, because you can get the side effect in the form of broken authorization. Alternatively, you can place the autoconfig file on your server: https://autoconfig.mail.ru/mail/config-v1.1.xml . In this case, your file will have a higher priority and you will be able to independently manage the authorization method not only at the testing stage. If your mail service has domain aliases, then you should not worry: the ISP server looks at the MX records. For more information about this server configuration method, see here .

2.5. Test build

Clone repository:

 hg clone http://hg.mozilla.org/comm-central cd comm-central python client.py checkout

Add a configuration for the test environment:

 ➜ echo $'ac_add_options --enable-application=mail\nac_add_options --enable-debug' > .mozconfig

We collect the project:

 ./mozilla/mach build

If during the build an error appears that the source code is outdated, then run the following command and restart the build again:

 ./mozilla/mach mercurial-setup --update-only

For more information about the build project, see here .

Possible problems:

For OS X 10.9–10.10 (at 10.11 this option interferes with the build) you may need to add the following option:

 echo 'ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/'

Also during the build process you may be asked to install autoconf 2.13:

 fx-team ./mach build /usr/bin/make -f client.mk -s Adding client.mk options from : MOZ_OBJDIR=/Applications/MAMP/htdocs/fx-team/obj-x86_64-apple-darwin14.0.0 OBJDIR=/Applications/MAMP/htdocs/fx-team/obj-x86_64-apple-darwin14.0.0 /Applications/MAMP/htdocs/fx-team/client.mk:304: *** Could not find autoconf 2.13. Stop. make: *** [build] Error 2 0 compiler warnings present.

OS X Solution:

 brew rm autoconf brew tap homebrew/versions brew install autoconf213

It may be our fault, but for some reason the configure file was generated with syntax errors:

 0:00.70 *** Fix above errors and then restart with\ 0:00.70 "/opt/local/bin/gmake -f client.mk build" 0:00.71 /comm-central/client.mk:361:      «configure» 0:00.71 gmake[1]: *** [configure]  1

Decision:

 rm configure && ./mozilla/mach build

Such an error indicates the absence of a YASM compiler in the system.

 0:09.52 configure:21252: checking for CoreMedia/CoreMedia.h 0:09.52 configure:21262: /usr/bin/clang -E -Qunused-arguments conftest.c >/dev/null 2>conftest.out 0:09.52 configure: error: yasm is a required build tool for this architecture when webm is enabled. You may either install yasm or --disable-webm (which disables the WebM video format). See https://developer.mozilla.org/en/YASM for more details. 0:09.52 *** Fix above errors and then restart with\ 0:09.52               "/opt/local/bin/gmake -f client.mk build" 0:09.52 /comm-central/client.mk:361:      «configure» 0:09.52 gmake[1]: *** [configure]  1

Solution (for OS X):

 brew install yasm && ./mozilla/mach build

Information on possible build issues can be found in the client.mk file. Be prepared for the fact that the source code of the project and the assembly will occupy 8.3 GB of disk space!

Launch of the project:

In the configuration, we specified the key - --enable-debug , it will help us to see all debugging information, including outgoing requests to third-party services.

 ./mozilla/mach run

The run command will automatically find the path to the application and launch it. In our case, after building the application is located in the following way:

 ./obj-x86_64-apple-darwin15.0.0/dist/DailyDebug.app/Contents/MacOS/thunderbird

Automated Testing:

To automate testing, Thunderbird uses the MozMill framework and XPCShell . We run unit tests:

 ./mozilla/mach xpcshell-test

For more information on unit testing, see the links below:

We run integration tests:

 ./build/pymake/make.py mozmill

For integration testing, the MozMill framework is used .

After the local run, the tests run the revision, and it also checks the declared functionality. As soon as the release engineer includes your patch in the release in the Treeherder CI , the regression testing cycle will start . For more information about other types (for example, testing for memory leaks ) testing, see this link .

A guide to Treeherder CI is here .

2.6. Testing functionality in early releases

As soon as the release engineer includes your patch in the early release, you can begin the next stage of testing. According to the work process, an early release called Aurora is going to be the first, then Beta and release. Links to download early releases are here . The calendar will help not to miss an important release date for you.

The general scheme of the stages of release is as follows:

The release cycle of each stage takes six weeks.

2.7. Backward compatibility

For customers who have not yet updated to the 45th release, the standard authorization scheme should work. And if you don’t think about it in advance, the user will always see an authorization error (unless manually changing the authorization method):

In order to maintain backward compatibility, we began to give the configuration file , focusing on the User-Agent:

 location /mail/config-v1.1.xml { if ($http_user_agent ~ Thunderbird/(\d|[1-3]\d|4[0-4])\.) { rewrite config-v1\.1\.xml /mail/original.config-v1.1.xml; } }

We set the title Vary: User-Agent

 add_header Vary User-Agent;

Now users of old clients will receive a configuration file without OAuth. Checking:

 ➜ curl 'https://autoconfig.mail.ru/mail/config-v1.1.xml' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 Lightning/4.0.5.2' 2>/dev/null | fgrep -i oauth

 ➜ curl 'https://autoconfig.mail.ru/mail/config-v1.1.xml' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/45.0.0 Lightning/4.0.5.2' 2>/dev/null | fgrep -i oauth <authentication>OAuth2</authentication> <authentication>OAuth2</authentication> <authentication>OAuth2</authentication> <authentication>OAuth2</authentication>

2.8. Release Testing

Now that a few long months are over, you can download the release from the main page ! Further we will show what the user will eventually see.

3. Usage scenario

There are several ways to add an email account to Thunderbird, but they all boil down to the same actions, so consider the most obvious:

1. Open the start page. In the section for creating a new email account, select Email :

2. We skip this step, since we already have a mail account:

3. Add a mailing address and click the button "Continue":

4. Select the mail collection protocol ( IMAP ) and click the “Finish” button:

5. At this step, we check the settings of the mail server and, if everything is in order, click the “Finish” button:

6. Enter the authorization data from your Mail.Ru account:

7. We agree that Thunderbird will collect mail from our account:

8. We are waiting for the letters to be downloaded:

4. Conclusion

As you see, we are trying to develop not only our opensource projects , but also third-party ones. We are extremely sensitive about security issues, so we decided to join the development of Mozilla Thunderbird and help with the implementation of OAuth 2.0. We hope our post will inspire someone to make their first pull request, and the world opensource article a little better.

5. Thanks

Kent James - for the code review and the inclusion of our patch in the early release.
Andrew Sutherland for helping with ISP.

6. Information links

Source: https://habr.com/ru/post/282319/

All Articles