Back in the distant past, when Nokia and Microsoft fueled the smartphones ball, one characteristic feature of mobile development emerged: the variegatedness of devices in terms of characteristics and modifications of operating systems. I had to test the application not only on different versions of the OS, but also on different physical devices. After the release of iOS, the phone models themselves have always been few, so there were no problems with them. But in the Android world, the problem of fragmentation has risen to its full height. There are thousands of models on the market, and new ones appear all the time, and your application or game should be guaranteed to work on each of them. Add more different versions of the firmware on these models ... And we’ll understand that you will need a lot of man-hours to check each release manually.

Note: we continue the series of publications of the full versions of articles from the magazine Hacker . Spelling and punctuation of the author saved.

Farm devices

Since the problem with the diversity of the fleet of devices appeared in ancient times, Nokia has already begun to offer its developers a service for renting smartphones . It was possible to put the application on the device and, in the remote control mode, to pass hands on key scenarios. It was free, and some devices had to wait a long time, but such testing gave the developer the hope that the software would work correctly on various smartphones and, therefore, users would have no complaints. The start was good, and the device farms were further developed - the OS itself began to provide tools to simulate user actions. That is, the tester or developer writes a script that pretends to be human. Add to this handy DevOps-tools for building and automatically running these scripts - and you can forget about that grim picture with a bunch of man-hours for each release.

In previous articles (the first and second , if you haven’t read them yet) we have already considered using mobile DevOps pipelines, so now we can focus on selecting a separate cloud farm of devices.

Today we will talk about popular cloud farm devices: Firebase Test Lab, Samsung Remote Test Lab, AWS Device Farm, Sauce Labs, Xamarin Test Cloud, Perfecto.

How it works

Built-in automation of UI testing appeared relatively recently: iOS 9.0 (XCTest UI) and Android 4.3 (UI Automator, although Espresso worked with Android 2.2).

At the lower level, everything is implemented quite simply: along with the application, a special script is launched that causes the OS to imitate the work of a real user: touching, stroking, gesturing with fingers and pressing the device buttons.

There are several popular subsystems for executing scripts: Appium, Calabash, Espresso, Robotium, UI Automator for Android, XCTest for iOS, which, in turn, support one or more programming languages ​​- Ruby, C #, Java, Python, Swift.

Free and angry

Since the problem with the heterogeneity of the fleet of devices is relevant primarily for Android, we will start the acquaintance with Google and Samsung farms.

Samsung Remote Test Lab

First in line we will have the Samsung Remote Test Lab service . This service is technologically outdated and would not be worth mentioning in our article, if not for one thing. Samsung is the leader and one of the legislators in the market of Android-smartphones, so early access to the flagship updates will allow you to test the performance of your application even before the appearance of devices on sale. Plus, there is access to Tizen-based devices: a line of Z smartphones and a Gear smart watch.

Working with the service is as follows: you reserve a device and launch a special Java application that provides remote access to the screen and input devices (touchscreen, buttons). Currently 25 models of smartphones and tablets are available, each of the models in several copies and modifications. Automation is done at the level of manual recording of the sequence of events, and the application must be installed by hand. In general, not so hot what opportunities, but completely free. And the most delicious - Samsung Remote Test Lab supports convenient remote debugging mode! So we can safely recommend this service as an additional farm for manual testing on Samsung devices.

Firebase Test Lab for Android

Our next service developed in the walls of Google is called Firebase Test Lab for Android . In general, Firebase is well suited to teams that specialize in Android development, and the device farm is just one of the tools. Currently, there are not so many device models available (about 30, the list below is in the screenshot), however, it is also possible to run on emulators. Test Lab is included in a single subscription to the Firebase service and can be completely free to start.

Firebase Test Lab, unlike the Samsung service, is easy to integrate into the DevOps pipeline. Test scripts can be implemented using Espresso, Robotium, UI Automator 2.0 and Robo tools. Screenshots are made during the execution of scripts. In general, this is a good solution for Android development of small projects using native tools. Cheap (free!) And angry.

Specialized professional farms

Android developers aren’t the only Samsung who are united, so I’d like to continue our review with larger farms that support iOS, have a large fleet of models and require money.

AWS Device Farm

Almost 400 devices (about 100 models) are available at AWS Device Farm , prices start at $ 0.17 per minute, there is anlim (!) And 1000 first minutes are free. It is worth noting the high quality of service and the ability to integrate into the DevOps pipeline. Appium (iOS + Android), Calabash (iOS + Android), Espresso (Android), Robotium (Android), UI Automation (iOS) and XCTest (iOS) and several others can be used to write scripts.

Xamarin Test Cloud

The next professional service is Xamarin Test Cloud . Over 2500 (no typo) real devices! Supports iOS, Android and a full set of features (screenshots, automated scripts, videos, promise also remote debugging and recording in the future). For everything about everything - from 99 dollars a month. The service is ideal for developers of cross-platform solutions (Xamarin, React Native), as well as for projects with a wide user audience (as a result, high coverage of the model range). Supports automated scripts based on Calabash and Xamarin.UITest.

Old men

And we will complete our today's review with two multifunctional services that also offer comprehensive services and services for testing a wide range of software: mobile and hybrid applications, as well as websites.

Sauce Labs

Meet this Sauce Labs . One of the old-timers of the automated testing market. Jason Huggins himself (the developer of Selenium) had a hand in his creation. Sauce Labs is an adult service for adult teams. Prices from $ 149 per month, there are native and hybrid applications for iOS and Android, and the ability to organize your private cloud or conduct testing in manual mode. There is support for integration with DevOps pipelines and running on emulators / simulators, although the device models themselves are stated to be no more than twenty. In other words, the fans of Selenium - the most it.

Perfecto

And we will complete our review of one of the oldest farm devices from the company Perfecto . Even in the days of Symbian and Windows Mobile, this company began to offer their devices for rent. Prices were high, but on trial it was possible to quickly run the application and make sure that it works (or does not work). About 20 different models are available for iOS, and more than 50 for Android. I suggest using Appium as a framework. You can manually test for free, but automation will cost from $ 299 per month.

Total

We reviewed the most interesting at the moment farm device, allowing to relieve manual testing itch. If you have not yet decided where to start, you can rely on the following:

• if you're alone and write in Java / Kotlin for Android, then boldly take Firebase Test Lab (for free);
• If you want to connect a remote debugger, only Samsung has it (for free);
• Looking for a service with maximum device coverage - we recommend Xamarin Test Cloud (from $ 99 per month).

In any case, there is a sufficient number of farms in the market for any technological stack and wallet, so you can easily find the right one. Now, finally, you can release the exhausted testers from the basement and return the passports to them. It's time to automate.

Before communication!

about the author

Vyacheslav Chernikov - head of development at Binwell , Microsoft MVP and Xamarin Certified Developer. In the past, he was one of the Nokia Champion and Qt Certified Specialists, currently he is the Xamarin and Azure platform specialist. He came to the sphere of mobile in 2005, since 2008 he has been developing mobile applications: he started with Symbian, Maemo, Meego, Windows Mobile, then switched to iOS, Android and Windows Phone. Articles Vyacheslav you can also read the blog on Medium .

Other articles by the author:

• 7 best device farm for mobile app testing
• OAuth Authorization for Xamarin Applications
• DevOps in the service of man
• Automate non-automated, or about Xamarin in real projects
• Convenient REST for Xamarin Applications
• Quickly create MVP (minimum viable product) based on Microsoft Azure and Xamarin.Forms
• Preparing Xamarin.Forms: Setting Up the Environment and First Steps
• Increase work efficiency in Xamarin.Forms
• Working with screen states in Xamarin.Forms
• Connecting Facebook SDK for Xamarin.Forms
• We connect VKontakte SDK for Xamarin.Forms

We remind you that this is the full version of an article from Hacker magazine .