The process of release of iOS-applications in Badoo

Hello! My name is Mikhail Bulgakov, and I work on the Badoo release engineers team. In this post, I will talk about how iOS application releases occur from the moment “I have a ready-made binary” to the moment “After the deluge of us,” and, of course, as we do in Badoo (looking ahead: we managed to reduce the time required to launch the release from several hours to one minute and get rid of manual work).

IOS application release life cycle

To begin with, I’ll briefly talk about how releases of iOS applications go in general. We assume that we have only one application with one supported language. Perhaps someone this part seems obvious, I understand. Therefore, if you know how this kitchen is arranged, switch immediately to the next chapter - iOS releases in Badoo.

Immediately I will clarify that all the work on the preparation of the release takes place on the site iTunes Connect. This is an Apple component specifically designed for application developers. There is both a web version and an application, and both have flaws. The application is almost useless for working on the product (but it should be at least for push-notifications: it is always better when they are, than when they are not); and the web version is cumbersome and cumbersome and slow (Apple is not particularly famous for developer-friendly products). There is also an API that can be used for our purposes, but more on that later.

So, when the application has already been assembled and tested, it is worthwhile to immediately upload it - it should go through the so-called processing (pre-check of a binary file), it takes from half an hour to several hours. Next, we prepare the information for display in the Store: fill in the text and image data that the user will see in the App Store. This data will also be checked: both the accuracy of the screenshots and the correspondence of the description of reality.

As soon as the binary file has passed the processing stage, we send the application for review by answering a couple of questions in the form (about encryption and advertising user ID), and we are awaiting a review. It is important to note that before the review has begun, we can change both text assets (“Description”, “What's New”, “Keywords”, etc.) and graphics (screenshots, icon and all that). But, as soon as the status becomes In Review , only text data can be changed, and even that is not all. On average, a review review lasts no more than a couple of days, the review itself lasts from half an hour to several hours.

If assets, in-app purchase (in-app purchases) or a binary file are not all right, then it is rejected and given the opportunity to correct errors. The most common ones are Metadata Rejected , when the reviewer could not use the new In-App, for example (in this case, you do not need to reload the new version - just chat with tech support and send the application for testing again) and Rejected (this is serious and you need to fix something). root and reload the new version of the application).

It is worth noting that the application can be rejected for so many reasons. Apple’s requirements are very specific and are detailed in the guidelines . And if we shake off Rejection, then we get a detailed description of the cause. These can be application crashes, inconsistencies between screenshots and applications, forbidden logic (there are many lists of logic that are not allowed to be distributed, for example, dating services for minors), a broken interface, and so on. In fact, in this case we get an additional stage of testing for our peace of mind, so you should not treat it too negatively.

There is also the possibility of an Expedited App Review . This is a very useful thing, intended for an urgent release of a critical fix or a seasonal event, to which we did not have time to prepare (for example, you need to add a new pack of stickers for Valentine's Day).

After checking the application, the status changes to Pending Developer Release , and we release it to the public.

By the way, there are also a couple of handy features for developers: automated / manual release of the application and Phased Review . As for the automated / manual release: you can release the application as soon as it passes the review, at a certain time and day, or manually - by clicking the button. And the second tool is a very interesting thing. Apple is launching a phased release (which was long ago implemented on Google Play), which allows the application to be released to all users within seven days.

Badoo iOS Releases

We now have eight iOS applications (the main one is Badoo, and several dating apps under different brands aimed at their target audience or country, with their own features and buns), most of them being translated into 25 languages ​​and dialects. For each application, we translate all materials for the App Store: as a text part, as well as screenshots and videos. Most of the applications we release once a week, the rest - periodically.

Naturally, for the preparation and release of such a volume of content, in addition to the binary file itself, you need to prepare and fill in a huge amount of texts and pictures. And, of course, it takes a lot of time from release engineers, translators and product managers. Without automation, all this would have resulted in a small local hell for all participants in the process.

Below, I will tell you how we went from this hell to a very convenient and transparent flow.

How was the release preparation?

While the QA team was testing the release branch, the product manager wrote the texts for the next release to Google Docs and sent them to the team of translators. Which, in turn, made translations in the same document. When everything was ready together with the binary file, the preparation of the next release began in the App Store.

The release engineer poured a binary file of the application in the App Store, waited while Apple finished its processing (at that time he was starting to upload updated texts and screenshots for the upcoming release). On average, processing took one to two hours. In about the same time, we managed to update the assets for the application.

Apple often added fuel to the fire with the godless iTunes Connect website, where all the work was done. And we actively used life hacking — listening to relaxing music. Because it was possible to fill in all the texts and screenshots, click on the “Save” button - and see that the session was torn or the message “Try later” (without trial!) Or unfilled assets for a non-existent language. By the way, such glitches are not uncommon now.

In general, it’s hard and unsightly to work with iTunes Connect if you have more than one application in several languages.

What were the obvious disadvantages of such an organization of work:

• Absolutely inconvenient system with a custom document for each release.
• Interaction through mail-jabber at each stage of release readiness.
• The document had to be searched each time among threads in the mail.
• Very slow and nervous work with iTunes Connect.
• Preparing each release took a few hours from the team.
• Huge probability of error due to the human factor.
• Writing texts for each release.

As previously monitored progress

Then began the process of monitoring releases. Yes, Apple sends emails about every change in application status. It's comfortable. Nevertheless, at this stage, it still required a lot of actions and manual work, which did not allow to reduce the probability of error due to the human factor.

Actually, earlier the review process on the Apple side took about five days. During this period, it was necessary to go into iTunes Connect from time to time and check if something had changed. The letter can be lost, and all this had to be done manually. And given the speed of the site, it was necessary to spend at least ten minutes to simply view the statuses of all applications. Just to view the statuses, Carl!

Finally, when all applications successfully passed the review, we notified product managers about this, so that they could be convinced of the readiness of the server part and send new versions “to float freely”.

Our days: what have we done to live better

Creation of the first interfaces

To solve the problem with custom documents, we first of all made an interface in which the product manager can upload a new version of the text through a special form. The changes immediately go to the translator's interface, which they use to translate both the main site and the application. –So we centralized the work of translators and reduced the number of custom entities.

In the same interface, you can watch and progress in all languages ​​for each application. Moreover, all translated texts also automatically fall into this interface. You no longer have to rush through a large document in search of the desired localization, and the number of errors due to the notorious human factor has noticeably decreased.

This was the first stage of automation. We have greatly simplified the lives of translators, product managers and release engineers. But we still had to copy all the texts in iTunes Connect.

Automation of preparation

At one point, a new tool loomed on the market, positioned as making life easier for developers of iOS applications — fastlane . We immediately started testing it. But at that time it was, unfortunately, a very raw product: there were a lot of errors and script crashes almost due to the phase of the moon. Nevertheless, he opened up many opportunities.

In short, the fastlane potentially allows you to automate almost all the work from compilation to release the application to end users:

• certificate generation ( Cert ),
• test user management ( Pilot ),
• application build ( Gym ),
• preparation for release ( Deliver ),
• the release itself and a whole lot more.

The appearance of this tool was all the more relevant because at that time we already had thoughts to write something similar on our own. But, as it happens, we didn’t get a hand and there was no time, because, in addition to iOS, we released apps for Android, Windows Phone, BlackBerry, Xiaomi and Opera and simultaneously worked on the desktop version. And at the same time it was necessary to support, improve and fix the flow of development, testing, deployment. But we started taking steps in this direction.

First of all, we taught our interface to unload all localizations of text assets, arranged in folders and files, in a single archive that can be used by fastlane.

Next, a script was written that prepared the file structure, initialized Deliver (a component of fastlane) repository, downloaded all localizations from our interface, put them where it should be and initiated the download to iTunes Connect through the same Deliver.

This script could be used by any product manager or tester, not just a release engineer. But the biggest advantage was that with the help of this script we reduced the time for preparing the store from half an hour to an hour to thirty seconds to a minute. Moreover, it allowed reducing errors due to the human factor to an absolute minimum.

After that, we thought that not every version needs a beautiful text. What's new, and every time writing bug fixes is not cool. Then we asked product managers to write 15 variants of standard texts, brought them into our interface (that is, they became possible to change, add to, delete without our team), and then taught our interface to replace the former text version What's new with one of the default options and set up ratification.

Now, if we do not have a very large release with new features and large-scale changes, we simply use one of the default previously translated texts. According to statistics, only one of ten to fifteen releases requires a lot of beautiful text. For other cases, the use of the default option (seemingly trivial) significantly saves the time and effort of all participants in the process.

It is also worth noting that Apple has a TestFlight component. It allows you to test new versions of the application on selected test users. After uploading a binary file there, it is in process for both the beta release and the main one. We do this immediately after the application is assembled, so that it has time to process itself before we are ready for release, which significantly reduces the waiting time for processing. That is, we save another half hour – hour.

Automation of progress monitoring

At the next stage, we decided to eliminate the problem of inconvenient and long-term monitoring of progress. At this point, Apple has significantly accelerated the review process: if earlier it took about five days to wait for the review, now it has taken only a couple of days. The smallest waiting period in our memory is a little less than a day (about twenty hours).

At that time, a custom script was already written that went to iTunes Connect with a curl and monitored application statuses. But because of the device site and the need to change the active development team (we also have many applications) in iTunes Connect, and also because of the periodic changes to the site, anything and anytime could break. Actually, that happened from time to time. In general, this option did not suit us due to the lack of time and hands for modernization, fixes and support.

Then it was decided to use the same magical fastlane. Fortunately, by that time it had become much more reliable and convenient to use. We wrote a script that every half an hour goes through the fastlane-Spaceship library in iTunes Connect, collects statistics on the status and active versions of all our applications and adds the results to the database. Based on this statistics, we wrote a user-friendly interface for product managers, where they can monitor the progress of the application from release preparation to release itself almost in real time, and also set up notifications and additional mailings to responsible employees.

Additionally, we made stickers on the page with a list of active builds with indicators for the release life cycle:

• Creating a release branch in the Git repository and testing.
• Preparing the stack for the upcoming release of the new version of the application.
• App submission and Apple review pending.
• If there are problems (rejections), then their conclusion.
• Successful completion of review of all applications and waiting for release.
• Successful release of all applications.

In addition to the obvious convenience for all participants in the development and testing processes, it allowed other teams to receive accurate dates for changing statuses, releases, problems, etc. The search for causes and the study of problems for absolutely all Badoo teams were also simplified.

So, now on one interface page, anyone can see the current progress in the release of absolutely all of our applications and the full history of all releases since the start of monitoring statuses and versions. And all this with exact dates of status changes, versions, etc. In the future, this will allow us to upload detailed statistics: how the time of a full release cycle for a month or year has changed, for example. And, of course, to make beautiful graphics with time, quantity, frequency - who does not like graphics!

What is the result?

1. The time spent on preparing and submitting one application was reduced (from an hour or two to (ideally) one or two minutes.
2. Manual work has been reduced to practically launching a single script.
3. A convenient system for monitoring the progress of the release from creating a release branch in the repository to the release of the application in the store has been developed.
4. It became possible to collect detailed statistics on the releases of all our applications and each separately.
5. Simplified the life of product managers, release engineers and translators.
6. Practically any participant can now make releases.
7. The process is almost completely integrated into the company's overall flow.
8. Our hands are completely free to upgrade release processes at other sites.
9. Almost complete care from manual work in the iTunes Connect web interface.

Future plans

We did a great job. But there are a few things that are being actively developed now and are in the plans for the near future.

Firstly, I want to make a convenient interface for filling localized screenshots with product managers. Due to the nature of such assets and the human factor, it is very difficult to automate completely. Fortunately, screenshots are rarely updated.

Secondly, I want to bring this whole process from launching a script to a release engineer or product manager to a single button in the interface by any participating participant in the process, which all QA and product managers use in one way or another. One click - and everything is ready for release! Lepote!