Tools for A / B testing iOS applications

In this article I will discuss several tools for A / B testing of mobile applications with examples and give their brief description. Successful testing also requires an analytics service where you can compare results. We will not consider independent services of analytics today, but we will touch on the opportunities that are inside the tools that will be discussed.

A / B testing of mobile applications

There are a large number of services that allow for A / B testing sites, but if you want to use this method on mobile applications, then there is also something to choose from. The required minimum for this kind of services consists of creating variables and distributing them among the audience. It is the responsibility of the application developers to interpret these variables correctly: direct replacement of textual and numeric data in the application, modification of the interface based on values.

Some services have a convenient visual editor that allows you to create various variations without changing the application. But the functionality of such editors is still very limited, with the help of them, for example, it will not be possible to change the interface of some screen so that with other model data this screen is displayed in a standard way. For the framework, this will be the same screen, and the changes will be applied everywhere.
')
We were faced with the task of comparing both text versions of variables (error texts, prompts) and changing the application interface (for example, having a sidebar and tabbar). From this proceeded in the compilation of our short-list of tools for A / B-testing. All services, which will be discussed further, provide an SDK for iOS and Android, with Windows Phone more difficult, in this case, the use of services is possible via HTTP requests.

Apptimize

apptimize.com

There are three ways to create a variant of A / B testing:

1. Visual (created using the visual editor).
   Allows in real time using the visual editor to change the attributes of interface elements (text in UILabel, image in UIImageView). The changed state will be a new version:
   
   
   
2. Code blocks
   Allows you to vary the execution of operations, for example:
   

   [Apptimize runTest:@"Dummy test" withBaseline:^{ //  } andVariations:@{@"variation1": ^{ //  }}]; 

   
   
3. Dynamic variables.
   Variants are formed depending on different values. Variables are declared outside the class using macros.

    ApptimizeString(name, value), ApptimizeInt(name, value), ApptimizeDouble(name, value), ApptimizeBoolean(name, value), ApptimizeArrayOfStrings(name, value), ApptimizeDictionaryOfStrings(name, value), ApptimizeArrayOfInts(name, value), ApptimizeDictionaryOfInts(name, value), ApptimizeArrayOfDoubles(name, value) ... 

   
   Alternate variable values ​​are set via the website. For example:
   

    ApptimizeString(screenName, @" "); 

   
   When creating options, it is possible to select a user base for testing.
   
   The framework allows you to log events:
   

    [Apptimize track:@"event name"] [Apptimize track:@"event name" value:<double value>] 

   
   It is also possible to aggregate events from other analytics services: Google Analytics, Mixpanel, Flurry, Localytics, Omniture.
   
   Price and number of users: there is a limited free version, $ 300 / month for up to 100,000 users, the above is not specified.
   

Optimizely

optimizely.com

The service provides the same three ways of creating options:

1. Visual
   
   
   
2. Code blocks
   

    [Optimizely codeBlocksWithKey:myCheckoutBlocksKey blockOne:^{ //   } blockTwo:^{ //   } defaultBlock:^{ //   }]; 

   
   
3. Variables
   

    OptimizelyVariableKeyForString(screenName, @"Hello"); 

   
   As you can see, the possibilities with Apptimize are identical. At first glance, the Optimizely visual editor is a bit less functional. Optimizely can be integrated with Fabric, but the service has a rather poor tool for analytics.
   
   
   Price and number of users: prices are not specified, there is a limited free version.
   

Mixpanel

mixpanel.com





Event Tracking:

 [mixpanel timeEvent:@" "]; //   [self uploadImageWithSuccessHandler:^{ [mixpanel track:@" "]; }]; 

Variables in Mixpanel are called tweaks (tweaks). To get the tweak, use the MPTweakValue macro:

 if( MPTweakValue(@"  ", NO) ) { //    } else { //    } 

There is a binding of values ​​for changing parameters on the fly:

 MPTweakBind(self.label, text, @"label text", @"Hello World"); 

It is used if, when the user is on the screen, you need to dynamically change the values ​​of variables (in other words, KVO is implemented for them).

The analytics tools in Mixpanel are quite good, it is possible to build complex filters, while using additional attributes sent from devices.

There is a visual editor , but it is not very stable and often “falls off”.



Price and number of users: you can choose the option by the number of users, or by the number of events per month. For example, 100,000 users will cost $ 250, or $ 350 for 2 million events.

Splitforce

splitforce.com

There is work with blocks :

 [[SFManager currentManager] experimentNamed:@" " applyVariationBlock:^(SFVariation *variation) { //  } applyDefaultBlock:^(NSError *error) { //  }]; 

And the ability to track the time of events:

 -(void)timedResultNamed:(NSString *)name; 

There is no visual editor , aggregation of analytics too. Built-in analytics is quite convenient.

Price and number of users: prices are comparable with other services, for example, $ 369 for 150,000 users.

Amazon

developer.amazon.com/appsandservices/apis/manage/ab-testing

The main advantage over analogues is its free. In order to use Amazon’s A / B testing functionality, you need to add an Amazon Insights framework to your project.

The SDK provides only “manual” mode of work with A / B testing projects:

 [self.abTestClient variationsByProjectNames:[NSArray arrayWithObject:@" "] withCompletionHandler:^(id<AIVariationSet> variationSet, NSError* error) { // -      variationSet }]; //   id<AIEvent> level3Start = [self.eventClient createEventWithEventType:@“ "]; [self.eventClient recordEvent:level3Start]; 

In my humble opinion, the interface is a bit primitive, but maybe it's for the best.

General conclusions

1. All services except Amazon are paid, which in some cases can become an obstacle to running A / B testing on a project.
   
2. All services are quite simple to use, implementation in an existing application takes minimal time.
   
3. All services operate asynchronously, so do not expect instant responses at low speed Internet connection.
   
4. Some of the services offer a visual editor. In practice, in my opinion, the thing is not so often used.
   
5. Those SDKs that represent access to variables through macros limit themselves to use. You cannot set a dynamic value inside a macro; therefore, you can get some values. Thus, A / B testing on the entities represented in the application by a large number is slightly difficult: either you have to list all possible variables, or get data on the entities in one variable and parse the value into components.
   

Summary table

Service Name	Ease of embedding (1-5 points)	UI editor (1-5 points)	Analytics (1-5 points)	Price per month	Support (1-5 points)	Overall score
Apptimize	four	five	four	$ 300 for 100,000 users	four	17
Optimizely	four	3	3	Not specified	3	13
Splitforce	3	0	3	$ 369 for 150,000 users	four	ten
Mixpanel	five	four	five	$ 250 for 100,000 users, or 2 million events for $ 350	five	nineteen
Amazon A / B Testing (Beta)	3	0	four	Is free	3	ten