Geekly Articles each Day
📜 ⬆️ ⬇️

Mobile App Auto-Test Tools Guide



... despite the fact that it is somewhat incomplete, contains a lot of doubtful or,
in any case, blatantly inaccurate, it has two important advantages:
firstly, it is a bit cheaper, [...], and secondly, on its cover large
and pleasant for the eyes written two words "Do not panic!"
- The Hitchhiker's Guide to the Galaxy

Hi, Habr!

My name is Arseny Batyrov, I work in the QA Badoo department and mainly do manual testing of web applications. I also conduct courses on manual and automatic testing of mobile applications.
')
Before launching a new course, I thought about what tools students should tell. Prosterstil Runet and the English-speaking Internet in search of comparative articles, but, oddly enough, did not find a suitable source of information. And then I decided to create it myself.

I pursued three goals:

  1. Classify tools on the auto-test stack so that their hierarchy and compatibility become clear.
  2. Show what tools are popular on the market today.
  3. Tell about the most popular tools of each type and compare them for several parameters.

The result of my work was this guide to the most popular and easy-to-learn tools for auto-testing mobile applications.

Use!


Content



Disclaimer: I am not a testing guru, and this article is not an exhaustive list. If you find a mistake or want to add a guidebook, welcome to the comments - I will definitely bring all the useful information from there with an update.

Application and Tests


First, let's understand what our tools will work with.

There are two important entities for us that are not part of the automation stack: this application and tests. The application is addressed by all automation tools. It interacts with the user and other applications through one or several interfaces: GUI, API, network interface, CLI and some others.



API (application programming interface) - the main interface for interacting with other programs.

GUI (graphic user interface) - a graphical interface, used to interact with the user.

Net (networking interface) - works through the network and is used by both advanced users and programs.

Tests can use all of these interfaces to interact with the application. In manual testing, the tester is the intermediary between the tests and the application: it converts the text of the test cases into actions with one of the application interfaces.



For automation, you need to replace the tester with tools that can interact with one or several interfaces of the application. Utilities for running and building a test suite will also be required.



Together, all of these tools are called an auto-test stack. To understand how they interact in the stack, it is necessary to classify them. The presented classification is conditional and necessary first of all for understanding the tools and their compatibility.

There are four groups of tools: drivers, add-ins, frameworks and combines. Consider them in more detail.

Tool classification


Driver


Auto-testing utilities, like other programs, can interact with an application only through a software interface — they cannot do otherwise. To work through other interfaces, there are special programs - drivers.

Driver - a program that provides an API for one of the application interfaces.



For each interface, except, in fact, the API, you need your own driver. For example, when you give the GUI driver the command “Press the Menu Button”, it takes it through the API and sends it to the application under test, where this command turns into a click on the graphic Menu button. Drivers are not needed or almost not needed to interact with the API of the application - software interaction. But when working with other interfaces can not do without them.

GUI drivers are usually the most complex, since this interface is very different from the usual communication program code. At the same time, in automated testing of mobile applications, the GUI is most relevant, since it is most often used in integration testing. The most popular GUI drivers for mobile testing are UIAutomator and Espresso for Android, XCUITest for iOS.

Superstructure


When the driver functionality is not enough or it is inconvenient and complicated, another level will appear above it, which I will call the add-in.

Add-in - a program that interacts with the application through one or more drivers, increasing the usability of their use or expanding their capabilities.



The add-in can have the following functions:
  1. Behavior modification (without changing the API).
    For example:
    • additional logging
    • data validation
    • waiting for action to take place within a specified time.
  2. Improving the convenience and / or level of API abstraction through:
    • the use of syntactic sugar - convenient names of functions, shorter references to them, a unified style of writing tests;
    • implicit control of the driver when, for example, it is initialized automatically, without the need to prescribe each such action manually;
    • simplifying complex commands like selecting an event from a calendar or working with scrolling lists;
    • implementation of alternative programming styles, such as procedural style or fluent.
  3. Unification API drivers.
    Here, the add-in provides a single interface for working with multiple drivers at once. This allows, for example, to use the same code for tests on iOS and Android, as in the popular Appium add- on .

Framework


On the other side of the tests is the launch framework. In this article I will briefly call it “framework”.

A framework is a program for building, running, and collecting the results of running a test suite.



The framework tasks include:

It can be noted that these functions are not related to testing only mobile applications - they can be successfully used in testing desktop and web applications. The fact is that the framework should not ensure the interaction of tests and applications - it works only with tests, and the type of application does not matter.

If drivers and add-ons are between the tests and the application, the framework is above the tests, organizing their launch. Therefore, it is important not to confuse the concepts of “driver” and “framework”. Of course, some frameworks have their own drivers for working with applications, but this is not a prerequisite. The most notable frameworks in mobile testing are xUnit and Cucumber .

Combines


Finally, another group of utilities used to automate the testing of mobile applications is combines that combine both frameworks and drivers (and not only mobile ones), and even development capabilities. Xamarin.UITest , Squish , Ranorex - they all support the automation of testing iOS, Android, web applications, and the latter two - also desktop applications.



So, we have classified the tools. It remains to determine the most popular in each category and compare them.

Poll


To identify the most popular and used utilities, I conducted surveys on several sites, communities, and channels for QA engineers, asking three simple questions. I did not limit the number of answer options so that I did not have to choose between different types of tools. It was also possible to add your own version - so there was quite a long "tail" of various utilities that do not fit into the classification. The results do not pretend to statistical accuracy, but they perfectly illustrate trends in the mobile application testing automation industry as of January 2018.







As can be seen from the results, the leading positions are occupied by utilities based on WebDriver: Appium and Selenium . Of the frameworks, JUnit and Cucumber are the most popular, and the second is more popular - it is surprising, because they still have different “weight categories”. Official drivers are more popular than unofficial ones for any platforms - apparently, due to high-quality support and more opportunities than third-party developments.

The three most used programming languages ​​look like this: Java , Python , Ruby (and Java leads by a wide margin). Ruby hit the top three I associate with the popularity of Cucumber.

Finally, the distribution by platform is quite expected - Android is ahead of iOS by a significant margin, then comes the Mobile Web . Were perhaps surprised that the answers about desktop applications for Windows in the last survey, but some combines allow testing mobile and desktop applications at the same time.

Having dealt with the popularity of tools, we turn to a comparison of the most significant. For each type, a comparative table of the capabilities of the tools that relate to it is first given. I tried to collect the most relevant and reliable information about each instrument, but could miss something. So if you suddenly find an error in the description, be sure to write about it in the comments.

Tool comparison


Drivers




In mobile testing, drivers are few and often they are developed by the same companies as operating systems. For Android, there are two official drivers: UIAutomator , which currently has version 2.0, and Espresso . Both are part of the Android Testing Support Library, developed by Google and well documented. In addition to them, there are projects Robotium and Selendroid , which are developed by third-party companies. All four products somehow work on the Android Instrumentation Framework - the basic API that Android provides for interacting with the system.

First, let's look at the drivers from Google. Both tools can work with WebView and hybrid applications, both support development in Java and Kotlin and work with emulators as well as with real devices.

Uiautomator


UIAutomator supports Android versions starting from API level 18 (Android 4.3). It does not require the implementation of its code in the project, that is, it can interact with already compiled applications. Moreover, when working with UIAutomator, you can use the full capabilities of the Android system: for example, turn on geolocation, call the system application, turn the device, click the Home button, or take a screenshot. Therefore, this tool is often used for functional end-to-end testing, either alone or with add-ins.

UIAutomator does not have its own recorder for tests, but there is a utility UI Automator Viewer that allows you to get data about the elements of the application running on the emulator or the real device and shows the locators of these elements. The hierarchical structure of all elements is displayed here, which is very convenient for using them in tests.



Espresso


Espresso, in turn, was intended more for white-box testing and was created as a tool for developers. It supports older APIs starting at API level 10 (Android 2.3.3), but requires access to the source code to run. Accordingly, Espresso cannot work independently with other applications and the Android system. But this tool has a recorder with which you can record simple scripts and use them at the initial stage of automation.

In general, if you want to test only the application, without taking into account its interaction with the system, and there is a desire and opportunity to work with the sources, it is better to use Espresso. In addition, it implements useful functions like automatic synchronization of tests with the application's UI, and you can not write various wait-commands.

If you need to test the application in conjunction with others or with system functions, and there is only access to .apk, choose UIAutomator.

By the way, these tools can be used together, because they are parts of the same library. Even in one test, you can combine the commands of both instruments.

Selendroid and robotium


Both Selendroid and Robotium were released before the official drivers appeared and still exist.

Robotium supports Android API versions starting from API level 8 and can work with WebView starting from API level 15. Selendroid works with a limited list of API versions - from 10 to 19. Both tools can access only one application, do not require access to the source code and support work on emulators and real devices. For Robotium, tests need to be written in Java, and Selendroid supports the WebDriver protocol, which makes it possible to use almost any popular programming language.

Selendroid has the Inspector utility, with which you can view a hierarchy of elements and record simple record-and-playback-tests. And Robotium provides the Robotium Recorder plugin for IntelliJ IDEA and Android Studio with similar functionality.

In general, these tools are developing much less actively than the drivers from Google, and their audience is much narrower. Nevertheless, from the survey results it can be seen that some companies still use them.

Xcuitest


In iOS, the UIAutomation driver was used for interaction with the application for a long time (which, among other things, caused confusion due to the similarity with the name of the Android driver), but since iOS 10 Apple stopped supporting this driver, and the XCUITest driver from the XCTest package appeared instead.

It supports iOS since version 9.0, and tests for it are written in Objective-C and Swift, like the applications themselves. To test an application, you do not need access to its code, and starting from Xcode 9, the driver can test several applications, including system ones, at the same time. “Out of the box” XCUITest allows you to run tests only on simulators, but with the help of some third-party utilities, you can get it to work with real devices.

XCUITest has its own recorder built right into the Xcode interface. With it, you can record simple UI tests, as well as find UI elements and their properties.



Add-ons




Appium


Appium is the most famous add-on today. It allows you to test applications almost regardless of platform, type and version of the system. Of course, this approach has several significant advantages and disadvantages.

Appium supports many drivers, not only mobile:

Support for such a variety of drivers is implemented in a rather interesting way: Appium uses the version of the WebDriver interface, which everyone knows from Selenium WebDriver. And, in addition to the large number of supported platforms, this approach has other advantages:


We are more interested in mobile application support, so let's dwell on the implementations of the UIAutomator 2.0, Selendroid and XCUITest drivers.

The simplest of them is UIAutomator 2.0, with which Appium interacts directly, passing the necessary commands to it. It works with versions of Android 5.0 and higher. From 4.2 to 5.0, you can use UIAutomator 1, and the interaction with older versions is provided by the Selendroid driver already known to us. To interact with XCUITest and work around some restrictions, use WebDriverAgent (WDA) from Facebook. WDA runs in the context of a simulator or a real device and passes commands through the API XCUITest.



Appium's disadvantages stem from its merits:

WebDriverAgent


Go to the WebDriverAgent add-in, which Appium uses to work with iOS. In fact, this is the implementation of the server side of the WebDriver protocol, which allows you to manage devices on iOS. Moreover, the functionality available is quite extensive: you can start and stop applications, use gestures and check the visibility of elements on the screen.

The add-in works with simulators as well as with real devices. For the detection of elements there is an inspector interface that opens in the browser. The add-in itself is supported by the Facebook and Appium teams and is actively developing. However, it can also be used separately from Appium, if for some reason the latter does not suit you.

Calabash


The next most popular add-on is Calabash for Android and iOS. The tool was developed by Xamarin, but it stopped supporting it in 2017, and now it is supported only by the community.

Each OS has its own add-on - Calabash iOS or Calabash Android. Both support WebView testing and Ruby / JRuby languages. Calabash Android does not need access to the application sources, but Calabash iOS will need to be connected to the Calabash framework application code. Also, to work with the view outside the native iOS-application, an additional tool is used - DeviceAgent, which allows you to detect these views and interact with them. Theoretically, this means that you can test and WebView, but in practice it is better to limit those views that iOS produces for your application: various overlays for confirmation, sending emails and inserting photos. Calabash Android supports work with WebView, but on a rather limited scale: tap, text input, alerts.

In general, Calabash is a fairly stable and fast tool that has useful functions to bring the application to the desired state (backdoors) and “out of the box” that supports integration with Cucumber. But due to the lack of official support, there may be problems when using it, and the community cannot guarantee their quick resolution.

Earl gray


Earl Gray is a kind of Espresso implementation for iOS, and it was also developed by Google. Everything here is standard for iOS add-ons: it must be added to the project in Xcode, tests can be written only in Objective-C and Swift, the application can be tested only one thing - it does not see external views. But testing is supported on real devices. By itself, the add-in is interesting, maintained more or less regularly, but for some reason it is not popular with testers.

Frameworks




The frameworks are least connected with testing mobile devices - they work with tests and integrate with any drivers and add-ins. Therefore, I will not consider them in detail (hundreds of materials on the Internet are devoted to this), but I will only make a superficial comparison.

xUnit and TestNG


The most popular xUnit frameworks. They were created as unit-testing tools, and the first such service was JUnit. At the same time, they can work not only with unit tests, but also with any others. Due to its versatility, xUnit frameworks are used everywhere and dominate web application testing. JUnit works only with Java, but now there are implementations of such frameworks for almost any popular programming language.

The TestNG framework, in which there are more various auxiliary functions, differs from this group somewhat.

Cucumber


Also popular are BDD frameworks, especially Cucumber. Unlike xUnit and TestNG, here tests and their steps are formed on the basis of documentation and are written in Gherkin, a language close to natural. I’ll clarify that Cucumber is still focused on acceptance testing, and it’s quite difficult to implement functional testing automation.

Combines




Xamarin


Xamarin is a service for mobile application development and testing, which has its own farm with mobile devices and tools for test automation, including on these farms. Development is conducted mainly in C #, there is a personal recorder.

Ranorex


Ranorex is a tool for automating almost any application. Able to integrate with Selenium, test mobile applications on emulators and real devices. Available only for Windows, using C # and VB.NET as a language for tests. Also has a recorder for tests.

Squish


Squish - also knows how to automate web, mobile and desktop applications, supports BDD, has its own recorder and IDE. You can use Python, Perl, JavaScript, Tcl, or Ruby to write tests.

The main advantage of such solutions is a complete testing cycle; there is no need to configure separate utilities and their interaction. However, all these tools are paid, often with closed code and rarely used for testing mobile applications. Therefore, I definitely cannot recommend them.

Conclusion


This, of course, is far from a complete list of possible functional testing tools for mobile applications. KIF, Frank, SilkMobile, TestComplete and many other utilities were left out of this article. The article was conceived as a guide to the main tools, and I hope that it will help someone to understand the auto-testing stack of mobile applications and not to be mistaken in the choice of services. If you have something to add on the topic, write to the comments, I will definitely read them and add some interesting things to the article.

For your convenience, I placed all the tools in one table and made a list of useful links - these materials can be found in the “Cheat sheets” section below.

Thanks


Many thanks to the whole Badoo team for their help in preparing and reviewing the article, you are great! Special thanks - z3us , nizkopal and Viktor Karanevich.

Cribs









useful links


Podcast with the exchange of experience in automating mobile application testing
Automation course for which everything was started
UIAutomator 2.0:
github.com/appium/appium/blob/master/docs/en/drivers/android-uiautomator2.md
bitbar.com/how-to-get-started-with-ui-automator-2-0
developer.android.com/training/testing/ui-testing/uiautomator-testing.html
XCUITest:
github.com/appium/appium/blob/master/docs/en/drivers/ios-xcuitest.md
developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/09-ui_testing.html
Espresso:
developer.android.com/training/testing/espresso/index.html
developer.android.com/training/testing/ui-testing/espresso-testing.html
Robotium:
github.com/RobotiumTech/robotium
github.com/RobotiumTech/robotium/wiki/Questions-&-Answers
Selendroid:
selendroid.io/setup.html
selendroid.io/faq.html
Calabash iOS:
github.com/calabash/calabash-ios
github.com/calabash/calabash-ios/wiki/DeviceAgent
Calabash Android:
badoo.com/techblog/blog/2017/01/24/break-limitations-with-calabash-android
Earl Gray:
github.com/google/EarlGrey
bitbar.com/how-to-get-started-with-earlgrey-ios-functional-ui-testing-framework
Appium:
appium.io/introduction.html
github.com/appium/appium
WebDriverAgent
github.com/facebook/webdriveragent
www.mutuallyhuman.com/blog/2017/04/20/webdriveragent-getting-started-with-automated-ios-testing
Cucumber:
cucumber.io
xUnit
junit.org/junit5
Testng
testng.org/doc
Xamarin.UITest:
developer.xamarin.com/guides/testcloud/uitest
developer.xamarin.com/guides/testcloud/uitest/intro-to-uitest
developer.xamarin.com/guides/testcloud/introduction-to-test-cloud/#The_Anatomy_of_the_Test_Cloud_Framework
Squish:
doc.froglogic.com/squish/6.0/tutorials-iphone.html
doc.froglogic.com/squish/6.0/tutorials-android.html
Ranorex:
www.ranorex.com/help/latest/android-testing
www.ranorex.com/help/latest/android-testing/automation-of-system-apps
www.ranorex.com/help/latest/ios-testing

Source: https://habr.com/ru/post/347986/

More articles:


All Articles