Features of hypothesis testing for mobile applications

How long does a hypothesis test for a mobile application take? Let's count:

1. Development of an application that works in different modes for different groups of users.
2. Testing the result.
3. Putting the app to app stores and waiting for approval.
4. Waiting for users to update the application. In 2019, auto-update is included for most, but not everyone.
5. Collection and analysis of statistics.
6. Bringing the application to the state of the winning hypothesis, in parallel with the development of the following ...

If your developers work on Scrum with two-week iterations, it usually means that testing a hypothesis takes a full month. With other methodologies, this period can be shrunk, but only slightly.

This state of affairs makes it impossible to achieve the rhythm of “5 hypotheses per week”, which many product teams are striving for.

Below I will explain how to speed up and improve this process and point out a number of ready-made solutions that can be used.

Go.

Turn on and off

Before you dive into the details, you need to enter an additional term - the Feature flag (Feature toggle) pattern .

For readers without a technical background, explanations may be required:

When developing a new feature, the programmer injects a “switch” in the application code that activates this feature. Typically, this solution is used to keep unfinished features in the off state in the general program code, but, of course, it can also be used to test hypotheses.

To use the Feature flag pattern in experiment verification, you will need:

1. Fully developed functionality on which to conduct an experiment.
2. The switch for it is in the default state “Off”.
3. Remote control switch from the server.

The question arises, on what time is saved, if the functionality still needs to be developed before you conduct A / B tests? Let's try to sort through the stages of the experiment:

What do we see here?

First, using the Feature flag, we can upload the application to the app stores until full testing for errors. We just need to make sure that when the new functionality is turned off, the application behaves as before - and this can be done with previously written autotests. The rest can be tested while the application is distributed to users.

Secondly, after completing the experiment, you can use the Feature flag to enable / disable functionality for all users until the next version is ready, where the flag will not be used anymore.

It is on this principle that the Apptimize service works , which provides a complete A / B testing system.

We analyze

To conduct an experiment, you need to do a few things:

1. Select a user segment if the experiment is not for everyone.
2. Select the size of the audience.
3. Collect data, and not only those that are verified by experiment. The remaining business metrics will be required to make sure that the experiment breaks nothing in other indicators.
4. Collect and analyze the result.

If you do not use a ready-made solution from Apptimize, the simplest approach would be a bunch of Google Analytics for Firebase for analytics and Firebase Remote Config for specifying individual configurations (segments and tests). These tools are adapted for collaboration.

Accordingly, you need:

1. Use Google Analytics for Firebase to track business metrics.
2. Use Firebase Remote Config to manage Feature flags.
3. Use Firebase Remote Config to set segments and experiment parameters.
4. Analyze data from Google Analytics using the keys from Firebase Remote Config in the analysis. This feature is provided by these out-of-the-box tools.

Optimize further

We discussed how to shorten the hypothesis testing cycle for mobile applications, reducing the time for testing and dissemination of the experimental results. But this approach does not allow to get rid of the time for approval and distribution of the application. The goal of “5 hypotheses per week” with this approach is still a little realistic.

To speed up the experiments, you need to be able to develop and submit new functionality without having to update the application. This can be achieved by using a dynamic user interface. However, this approach has problems:

On the one hand, there are technical restrictions on the construction of the interface on the settings received from the outside. Most mobile development frameworks use a declarative approach, where it is impossible or very difficult.

On the other hand, application store policy prohibits the loading and execution of arbitrary code, since it can be used for functionality that violates the rules of application stores.

Another limitation is the amount of data transferred from Firebase Remote Config. It cannot be used to transfer the entire interface. It is optimal to store in it only the “key” of a specific version of the interface, and when changing this code, load the interface from a third-party service. In itself, it does not limit the choice of mobile development framework, but requires additional implementation efforts.

The optimal solution is an approach in which only the user interface is dynamically built, and the business logic remains fixed. Since the vast majority of product experiments concern the interface, it is possible to maintain a high rate of work. At the same time, experiments that require the refinement of business logic can be performed in parallel, according to the process described above with flags.

Technically, this approach is easiest to implement in a framework that has the following characteristics:

1. A reactive high-performance user interface that does not use the default declarative approach.
2. Support for Google Analytics for Firebase and Firebase Remote Config.
3. Desirable cross-platform solution, for the overall acceleration of development.

Optimally, these criteria are met by the Flutter framework. As a proof-of-concept of this approach, there is a library for it that allows you to create a dynamic interface .

Using the dynamic interface created in Flutter, Google Analytics for Firebase and Firebase Remote Config, you can develop applications that are easy to test hypotheses will be comparable to web sites.