Introduction

Every year it becomes more and more obvious that specialized services are needed to distribute test versions of iOS applications. This is relevant not only for companies developing applications on an “industrial scale”, but also for indie developers. The reasons are: an increase in the number of devices with a “non-standard” / ”non-multiple” screen size, differences in the architecture of the central and graphics processors — everything that used to be commonplace for Android development now becomes fair for iOS.

Historically, the most popular distribution service was TestFlight - almost ideal, both technically and in usability. However, after the acquisition by Apple in early 2014, the service underwent a number of ambiguous changes that, on the one hand, negatively affected its leadership position, and on the other, gave a chance to alternative services to fight for first place and offer at least an equivalent replacement, and maybe something more comfortable and functional.

In my series of articles, I am going to share the results of comparing several distribution services and as a result help others who are facing the same question of choice.
')
The following services participate in the review (parts of the review will be published in this order):

• Apple TestFlight
• HockeyApp
• Ubertesters
• Crashlytics (as part of Fabric)

TestFlight of the old version will not be considered, since although it still works and is available at www.testflightapp.com , using it in the long run does not make sense for a number of reasons:

• there are no guarantees that Apple will not pause or stop it at any time.
• SDK service is not available for download, moreover it is impossible to use the service if the application uses the SDK
• the service is no longer cross-platform, only iOS application distribution is supported ( more )

Important! Literally on the eve of the publication of the article, the TestFlight team (of the old version) sent out a notice stating that the old service would stop its work on February 26, 2015 ( details ).

Evaluation system: services will be evaluated on a 10-point scale for each of the sections (Registration and Integration, Main Functionality, Additional Functionality, Continuous Integration). The total score will determine the winner (the final conclusion will be included in the last part of the review).

Apple's TestFlight

Registration and Integration

To use the service besides the developer account (iOS Developer Program), you need to access iTunesConnect. The functionality depends on the role (group) to which your iTunesConnect account belongs. Users from the Technical group can:

• enable / disable TestFlight Beta Testing mode for applications (Prerelease tab)
• choose testers from the list of available Internal Testers, but do not have the ability to form this list on the iTunes Connect page -> Users and Roles -> TestFlight Beta Testers -> Internal
• create a list of external testers My Apps -> APP -> Prerelease -> External Testers (but to distribute versions for external testers, the application must be reviewed by Apple)

Users from the Admin group have the same rights as users of the Technical group, and can also add new users to the Internal Testers list.

At the moment, installing or using any specialized SDK is not required.





Rating: 8/10.

Main functionality

If we compare the new version of TestFlight with what was previously available to developers as part of the standard Apple services, then it is, in some ways, undoubtedly a step forward. For example, the number of testers is increased from the conditional 100 to 1125 (of which 100 are devices available for AdHoc distribution before, 25 are internal testers (Internal Testers), and 1000 are external Testers), and if earlier was carried out to the device, now the Apple ID is a unique identifier, which means that each tester can test the application on all devices available to it (and, of course, supported by the application).

Note (June 15, 2015): the number of devices available when using AdHoc provision profiles from June 8, 2015 was changed from 100 in total to 100 for each device family (the change occurred after announcing at WWDC that all developers programs were merged into one) .

The difference between internal and external testers is an additional step that is added in the second case: a test version of the application must be sent for review and approved by Apple experts (Beta App Review). Moreover, the initial review will be carried out more thoroughly, and all the latter can be faster (but not necessarily). This, of course, imposes certain restrictions on the development process and requires additional planning of the testing phases of the application, taking into account the uncertainty of the review deadlines (as is the case with the usual review of applications before publishing to the AppStore, if the publication is refused, the repeated application is considered again in the order of ).

But there are other features that distinguish Apple’s solution from the alternative, alas, not for the better:

• The service client application can only be installed on iOS 8.0 and higher, respectively, there is no way to test applications on devices with previous iOS versions.
• To distribute test versions, you must create an AppStore Distribution provisioning profile and set up an application profile in iTunes Connect.
• Since the download of the distribution package of the application is carried out through the standard Xcode wizard -> Archive -> Export, it is necessary that the application project contains 3 main icons.

• The number of applications that are simultaneously available for testing is limited to 10.
• The service is very slow, it concerns both the speed of updating the list of available applications and their versions, and the speed of downloading / installing applications on test devices.
• Sending feedback from the test device is carried out using the client application, but since this is an ordinary letter, there is no possibility to attach a screenshot (as a plus, it can be noted that a block describing the characteristics of the test device is automatically inserted into the letter).
• The ability to send a review appears only if there is an indication in iTunes Connect in the settings of what exactly you want to test (and if you do not specify an e-mail address for receiving reviews, the “To” field in the letter will be empty).
• Only one version of the application can be available for testing (the tester does not have a choice, the accessibility settings are made in the application profile in iTunesConnect).
• The “Feedback Email” field in the application version settings in iTunesConnect has to be filled each time (it would be more logical to remember, leaving the possibility for editing).
• As it is not surprising, but TestFlight in Apple’s version is not a 24x7 service, for example, at the end of December, when trying to access iTunesConnect, you could see the following message: iTunes Connect is unavailable until December 29. (To be fair, it’s worth noting that Apple arranges such “holidays” annually and sends a notification a week before their occurrence.

However, there are some unique features: for example, the service client (TestFlight application) is the only one who probably knows the installation status of the test version, alternative solutions simply do not have the ability to use system functions that allow you to implement this.

Note: installation status - a change in the installation process display, familiar to users of the AppStore. When the status alternately gets - install - installing - open. In other distribution services, this chain looks like this: Install - Installing - Install - applications simply do not know the result, whether the test distribution was installed or not. In the case of the Apple TestFlight, the chain has the usual look: Install - Installing - Open.



The icon of the updated TestFlight application (it would be more correct to say that this is a completely new native application developed by Apple). Below is an example of the test version of the application icon - the orange “dot” allows you to determine that this is not a version from the AppStore.



Apple’s “insurance” against spam charges.



Apple does not allow downloads of distributions without changing the version or assembly number.



Rather concise and clear administration page of application versions for testing.



An invitation letter for participation in testing, HTML formatting is not very optimized for mobile platforms (in fairness, I want to note that the Mail application displays such letters correctly, but letters from other services are displayed normally in OWA).



TestFlight application: information about an application available for testing.




The TestFlight application only knows how to determine the result of installing the test version (the Install button has changed to Open) - an advantage gained by moving to Apple.



An email with a feedback sent from the TestFlight application. In addition to poor formatting and the inability to add attachments, you can note another disadvantage: the sent reviews only go to the mailbox, this information is not stored in iTunes Connect and is not available.

Summarizing, we can say that the updated version of TestFlight just works. At the moment, it does not have any particular competitive advantages (we are talking about functionality, and not an indisputable advantage - belonging to Apple, which is still more potential).

Rating: 6/10.

Additional functionality

At the moment, unfortunately, Apple takes only a number, not a skill. From a clear advantage, it can be noted that a much larger number of beta testers can be used for external testing: an order of magnitude more than all other alternative solutions (the reason is clear: the company controls the distribution of applications and has an advantage over third-party solutions).

Update (May 5, 2015): Crashes Organizer became available to developers with the release of Xcode 6.3. The new functionality is undoubtedly a good sign that Apple continues to develop its services, but, unfortunately, the company still lags behind other services:

When you open the Crashes organizer, Xcode begins refreshing the crash reports for your apps. Xcode downloads the report crash reports — you’ve seen it during the past two weeks. However, there may be a number of times you can distribute your app and crash reports available in Xcode.

Three days of delay and a two-week period of archival access look somewhat discouraging.

Rating: 1/10.

Continuous integration

Note: The following nuances of using services as part of continuous integration (continuous integration) in this review imply that it will be based on the solution proposed by Apple. That is, with the help of Mac OS X Server and Xcode bots - in detail on the organization of such integration, I will not stop, perhaps it will be a topic for a separate article. Those interested can familiarize themselves with the topic, for example, by referring to Apple’s official guide.

No matter how strange it may seem, but at the moment there is no possibility to publish applications to Apple TestFlight in automatic mode. I hope that this big flaw will be fixed by the company in one of the next major Xcode updates, especially since the original TestFlight had such an opportunity.

Rating: 0/10.

Total total score for all sections: 15 points.

To be continued...