In the process of improving Mail.Ru Mail, we are constantly rolling out new functionality. Today I want to share with you the subtleties of how the product is updated with a multi-million audience, using the example of a complex and interesting task - redesign of the Mail. Looking ahead, I will list the stages of testing that a large and complex feature passes before implementation:

1. Small group (usually the developers themselves)
2. Ux lab
3. Company colleagues
4. Beta users
5. Users who enabled the update themselves
6. Split
7. All Mail users

Mail in a new design - inside we called it "Septima" - had to successfully go through all these stages.

When choosing a development method, we considered two options: program the product side by side or update the old code. The temptation to write everything from scratch was great, but it was important for us to send Septim to the first and second stages as soon as possible: after all, ideas on updating the design and changing the user experience should be tested in practice, and the only practice in our case is the daily use of the mailbox. If we had chosen development from scratch, we would start testing much later than in the variant with the modification of the old code. Given this, we stopped at the second method. By developing a new Mail on top of the old code, we were able to quickly implement the basic functions in the new design, which allowed us to quickly start using it.
')

Septim on developers (prototype)

It is important to understand that the cost of correcting a design error is multiplied with each week. If you write a product that nobody uses for half a year, your risks are person-months (and if there are several developers, then person-years). Moreover: in half a year, to understand that the key thing had to be done differently, it is equivalent to lagging behind the competitors by half a year.

In Septim, we changed the appearance and behavior of the toolbar and focused on getting the Mail with a new toolbar as quickly as possible and trying it out in action.

The bottom line: having received a prototype, we were able to touch what previously existed only on layouts. This is extremely important for making further decisions.

Ux lab

Testing the prototype, the developer looks at him with a long-awaited look: after all, he knows about the new changes long before he tries them. Users have a fresh look, so it was important for us to test in the laboratory and see their reaction to updates.

For testing, we selected the main functions of Mail (write a letter, open, reply, etc.) and asked users to perform these actions. It was important to make sure that for users these actions remained as simple and intuitive as before. The changes should not be confusing; Moreover, they had to make daily activities easier.

Bottom line: The UX-lab helped us make sure that the direction was chosen correctly, and we can move on.

Corporate users

In terms of testing, the difference between colleagues from other departments and other users is not very large: for them, changes are also unexpected. The only fundamental difference is that colleagues can promptly report a problem or leave feedback. An important point. With each significant feature we roll out a form with a vote and the ability to leave feedback about this feature. We consider the figure of 75% or more positively positive to be good.

Bottom line: rolling out colleagues helped us solve two problems - get the first mass reviews and check the product for defects. The reviews turned out to be more than positive, we have eliminated the defects that have been reported. It was possible to move on.

Beta users

Updating Mail.Ru Mail is a huge load for testing: autotests are not written yet, and manual testing cannot grow several times in a short time. Beta users are required in such situations. It is very important to choose them correctly. We decided that we need sufficiently advanced users who are not afraid of our proposal to participate in beta testing, as well as be able to clearly explain what problem they faced (at least know the word "browser", "version", "OS" and t .P.). In addition, they should be quite loyal users who are accustomed to our interface. We came up with the following criteria for "advancement":

• use Chrome or Firefox
• created more than three folders in the Mail
• use our mobile application

Those who met all the listed criteria, we showed a plate with a proposal to sign up for beta users. We offered them to test a new design with the ability to turn it off. The result was an even greater amount of feedback and extensive testing of the Post with high-quality reports about problems. Interestingly, some reports were immediately with screenshots of the console. And one was a piece of CSS-code, which was proposed to add to the project.

Bottom line: in a review, users reminded us of several features from the old Mail that we lost sight of in the new version. A vote once again confirmed that people like the new product.

Self enable

For the next stage, we made a "smart" letter. As soon as the user entered the Mail.Ru Mail web-interface, he received a letter with a description of Septima and the "Enable" button. If the user opened the letter in Septim, the button in the letter changed to “Leave a comment”. On average, 30% of those who read the letter switched, which in total with the other restrictions gave 11% of the entire audience.

An interesting fact: for the sake of experiment, we made two buttons for switching to the new interface: “Update” and “Update to version 7.0”. The first CTR was 5% higher.

At this stage it was important to understand the reaction of users. The number of users (90,000 voters) and reviews (52,000, all reviews read and processed) already allowed collecting accurate statistics. To our joy, only 2% of those who voted were against the new design.

This stage was difficult for the team, since with such a number of users rare system configurations begin to appear (I’ll make a retreat: the configuration includes the operating system, a browser of a certain version, plug-ins to the browser, antivirus, network environment, etc .; each of these items , as well as their combination, can cause problems with your product). Therefore, rolling out to a significant percentage of the audience requires compliance with several conditions: this should occur at the beginning of the week, you need to make sure that all key developers are not sick and not on vacation, and the support service is on alert. There are several tense days ahead: users start reporting problems, and you need to use the whole arsenal of tools to identify them - server logs, reports from browsers, individual logging, and even Team Viewer, if the user does not mind.

Bottom line: in our case, the main problems were brought by third-party browser plugins, which counted on the old layout of the site. For example, NCH Toolbar broke a scroll in Mail, and WebBars - writing a letter. Representatives of the whole family of plug-ins (ExSmile, SmileEx, MegaSmiles) in addition to breaking the display of the Mail, also turned out to be Trojans, illegally infiltrating computers and user browsers.

After all the problems were resolved, we were ready for the next stage.

Measure retention

Reviews by reviews, but before switching the main mass of people, we needed to make sure for sure that the new design works, at least as good as the old one, and we don’t lose users who switched to it. A good indicator for such a case is user recurrence. To evaluate it, we divided users into three groups that are the same for all parameters. To the users from the first group, we switched the interface and showed the tutorial on Septim, which told about the main changes in the Post and morally prepared for the fact that the familiar interface would change. The second group was simply switched the interface, and for the third everything remained as before. The highest return rate was among the first group of users.

Bottom line: we made sure that the new interface not only does not deter old users, but also increases their return to Mail.

All users

If you launched changes to the current functionality, then you know that the biggest enemy is “Return everything, as it was, why did you change everything?” The most favorite reading I read is “Remove your hands from our site!” And it doesn't matter what as a result of the update, we have reduced the number of clicks from five to two: for the user, this is his own five clicks, he is used to them.

Everyone wanted to roll out a new interface in the same year in which they started to do it. And we were ready on all counts. All monitoring indicators — download speed, number of letters sent and read, return to Mail — were either higher or the same as in the old Mail. It remained to measure another indicator - how much the users liked the interface. At that time, the positively rated Septim was 98%, but it was necessary to take into account that this was the opinion of users who themselves decided to switch. We "pulled the switch" - switched all - and then again measured the ratio of those who were "for" and "against."

Bottom line: at the moment, the share of those who like Septim is 93% - this is a new record for us.

If you have had experience of large-scale updates, I will be glad to hear about it in the comments: are you trying to minimize the stress associated with the transition to the new version for users, how you test updates, what steps you share in rolling out updates?