Automation testing of mobile applications: a comparison of tools
Test automation helps solve several problems at once - including when it comes to mobile applications. Instead of manually performing routine, time-consuming procedures, specialists can delegate a significant part of them to frameworks. Automation simplifies testing and helps speed up regression testing, and also makes it possible to use previously unavailable test types.
We will compare several tools that have proven themselves in the market and continue to develop. This knowledge will help you choose which solution to use to test a particular mobile application.
')
This article is unlikely to open up new horizons for professionals, but it may be useful for beginners who have decided to master the basics of mobile testing, and to some extent to mid-level specialists.
The first thing to build on is the platform on which the application runs. Based on this, we classify the list of tools as follows:
Android
iOS
Universal
Powerful testing tool with advanced external integration. This means that this framework not only allows you to test the application itself, but is also able to “communicate” with the operating system and other applications — for example, activate and deactivate Wi-Fi, GPS, open the settings menu during the test and produce other external interactions.
The purpose of UI Automator is to test black-box testing. This means that the analysis is performed from the standpoint of an external user without access to the code.
The main features include:
Link to the documentation .
A lighter compared to UI Automator tool, not suitable for interacting with external applications, but convenient for testing the “white box” (white-box) with access to the source code of a specific application or testing the “gray box” (gray-box) with which has access to some internal processes and structure.
At the same time, Espresso stands out with its powerful API https://github.com/hamcrest . The interface adds convenient methods for checking in auto tests, for example:
assert_that (1, less_or_equal (2)). Special methods are used to test the webview.
UI Automator and Espresso mutually complement each other and can be used in combination within the same project.
Link to the documentation .
A tool for black-box testing without reference to the application code. It works only with native products - unfortunately, it will not work to conduct cross-app tests.
On the other hand, the native nature of the framework is an advantage from the point of view that when using XCUITest the degree of mutual understanding of developers and testers is at a much higher level than in cases where one and the second use different languages.
A useful addition is the test recorder, which makes it possible to write tests by recording actions in the application, even to those who do not work with the code.
The tool allows you to avoid common mistakes and unnecessary, inaccessible to the user code manipulations. However, XCUITest has some drawbacks.
XCUITest, unlike Espresso, works in a separate thread, during testing you need to wait for the appearance of certain elements and parameters. The current state of the application is not read, and delays in updating the data may make it impossible to detect the requested items.
Documentation XCTest and XCUITest .
EarlGrey focuses on reproducing the user experience. While the elements on the screen are not presented visually, the simulation of work with the application does not start.
There are a number of amenities and benefits. First, specialists like how the framework synchronizes requests, UI and threads. No waitforview and wait are needed.
Secondly, as already mentioned, special attention is paid to tracking the visibility of elements. The tool has an additional layer for checking the load of the interface and reproduces user gestures - swipes, clicks - directly at the application event level.
Repository links: github.com/google/EarlGrey and google.imtqy.com/EarlGrey .
Universal tools (or “combines”) make it possible not to limit your choice only to Android or iOS, but to work with both platforms.
Such tools are applicable for testing applications of the following types:
In our opinion, Detox is convenient for applications written in React Native. Tests are written in JavaScript, while iOS and Android applications are generated from the same JavaScript code and are as similar as possible. This allows you to use the same tests for both platforms.
The key feature of Detox is gray box testing. In this case, the framework has some access to internal mechanisms, which allows us to relate the external behavior of the application to what is happening at a deeper level.
Detox can access memory and monitor running processes. The principle of gray-box helps to deal with the instability, which is expressed in the fact that with end-to-end testing:
Oddly enough, the “gray box” shows not only better stability, but also a higher speed than the “black box”. Avoiding all sorts of pauses, waitUntil, gray-box can be 5-10 times faster.
Detox does not need WebDriver, working with the native driver via JSON. It uses native methods directly on the device. Inside this framework, apply EarlGrey for iOS and Espresso for Android.
The framework works with both emulators and physical devices.
Link to the documentation .
The advantage of Appium is that it is possible to write tests for each of the platforms using a single API, without resorting to converting the application to any particular framework compatible with the framework.
When testing frameworks from vendors are used - that is, you work with the original application. For Android 4.2+, respectively, UiAutomator / UiAutomator2 is used, and for iOS 9.3+ - XCUITest. WebDriver (also known as Selenium WebDriver) is used as a framework framework.
Principles of Appium:
Using Appium is justified when you need a tool to automate testing on multiple platforms at once. It is useful if you have specialists with experience in testing web applications, but no experience in automating the testing of mobile applications.
In general, it is a flexible tool that can be modified to the needs of the project without the need to adapt to a limited set of development languages.
Link to the documentation .
Paid comprehensive tool for testing desktop, mobile and web applications. Allows you to test as with the use of programming, and without the use of scripts. Provides the ability to test not only through emulators, but also on live devices.
The tool allows you to create and configure tests, as well as manage them centrally. You can create a test in the control center and run it in various external environments and on any devices.
It is easily integrated with the existing CI environment: with application management systems such as Jira and TFS, as well as with version control systems, such as Git and SVN.
In Ranorex, data-driven testing with data loading from SQL, CSV, Excel has been improved.
The tool is absolutely suitable for any device, supports parallel testing on each of them.
Combines all three approaches to testing: black-box, white-box and gray-box.
Link to the documentation .
Paid environment for automating the testing of mobile, web and desktop applications. It supports Android and iOS, and in terms of application types: native, web-based applications and hybrid.
Focused mainly on functional and unit testing, the tool also provides the ability to perform many other types of testing:
In TestComplete there is a Recorder - in it tests are created by recording actions and setting up commands in the editor. Then they can be launched either directly in the tool itself or exported to third-party applications.
This tool recognizes objects and controls, offering special commands to emulate user interaction with them. Integrates with Jenkins, Git and Jira, which allows you to run continuous seamless testing.
Link to the documentation .
Planning to test this or that mobile application, pay attention to the tools listed above. Each of them has its own characteristics, and sometimes limitations.
Consider an example. If you are faced with the task of testing a small application in a short time, first of all you need to take into account such factors as the type of application being tested and the experience of your specialists. If the developer writes tests, it is better to choose a native language and a tool for its platform (see the table below). If the tests are done by SDET specialists who are familiar with other languages ​​(Java, JavaScript, Python, etc.) and have worked with Selenium, it is convenient to use Appium. If there is no experienced SDET in the team, and QA specialists will write tests, it is better to choose paid frameworks, since they have utilities for recording tests and more stable technical support than in open source frameworks.
From our practice:
We worked with one online store, which had two mobile applications - on iOS and Android. To cover the main user scenarios with tests, we chose Appium for several reasons:
As a result, Appium fully met expectations, we successfully conducted tests for iOS and Android. It should be borne in mind that similar end-to-end tests with Appium are not carried out at each merge request, since this takes a lot of time.
In conclusion, we offer you a table that will help you choose a tool for your project. It is worth noting that in some cases the division in the table is conditional. Somewhere for simplicity, a generalization has been made and only the most basic parameters are given. Testing tools are constantly evolving, so when choosing a framework, it is important to check the actual documentation.
Thanks for attention!
We will compare several tools that have proven themselves in the market and continue to develop. This knowledge will help you choose which solution to use to test a particular mobile application.
')
This article is unlikely to open up new horizons for professionals, but it may be useful for beginners who have decided to master the basics of mobile testing, and to some extent to mid-level specialists.
Tool classification
The first thing to build on is the platform on which the application runs. Based on this, we classify the list of tools as follows:
Android
- Espresso
- UI Automator
iOS
- XCUITest
- Earlrey
Universal
- Detox
- Appium
- Ranorex
- TestComplete Mobile
Android Application Automation Testing
UI Automator
Powerful testing tool with advanced external integration. This means that this framework not only allows you to test the application itself, but is also able to “communicate” with the operating system and other applications — for example, activate and deactivate Wi-Fi, GPS, open the settings menu during the test and produce other external interactions.
The purpose of UI Automator is to test black-box testing. This means that the analysis is performed from the standpoint of an external user without access to the code.
The main features include:
- UI Automator Viewer for tracking and analyzing the components displayed on the screen during the test. It provides information about the elements and their properties, which makes it easier to create more relevant tests.
- API for getting information about the state of the device and running processes on it.
- UI Automator APIs for cross-platform tests.
Link to the documentation .
Espresso
A lighter compared to UI Automator tool, not suitable for interacting with external applications, but convenient for testing the “white box” (white-box) with access to the source code of a specific application or testing the “gray box” (gray-box) with which has access to some internal processes and structure.
At the same time, Espresso stands out with its powerful API https://github.com/hamcrest . The interface adds convenient methods for checking in auto tests, for example:
assert_that (1, less_or_equal (2)). Special methods are used to test the webview.
UI Automator and Espresso mutually complement each other and can be used in combination within the same project.
Link to the documentation .
Test automation for iOS applications
XCUITest
A tool for black-box testing without reference to the application code. It works only with native products - unfortunately, it will not work to conduct cross-app tests.
On the other hand, the native nature of the framework is an advantage from the point of view that when using XCUITest the degree of mutual understanding of developers and testers is at a much higher level than in cases where one and the second use different languages.
A useful addition is the test recorder, which makes it possible to write tests by recording actions in the application, even to those who do not work with the code.
The tool allows you to avoid common mistakes and unnecessary, inaccessible to the user code manipulations. However, XCUITest has some drawbacks.
XCUITest, unlike Espresso, works in a separate thread, during testing you need to wait for the appearance of certain elements and parameters. The current state of the application is not read, and delays in updating the data may make it impossible to detect the requested items.
Documentation XCTest and XCUITest .
Earlrey
EarlGrey focuses on reproducing the user experience. While the elements on the screen are not presented visually, the simulation of work with the application does not start.
There are a number of amenities and benefits. First, specialists like how the framework synchronizes requests, UI and threads. No waitforview and wait are needed.
Secondly, as already mentioned, special attention is paid to tracking the visibility of elements. The tool has an additional layer for checking the load of the interface and reproduces user gestures - swipes, clicks - directly at the application event level.
Repository links: github.com/google/EarlGrey and google.imtqy.com/EarlGrey .
Universal tools
Universal tools (or “combines”) make it possible not to limit your choice only to Android or iOS, but to work with both platforms.
Such tools are applicable for testing applications of the following types:
- Native applications (native apps) - written directly under Android, iOS and Windows SDK.
- Mobile web apps (mobile web apps) - accessible through a mobile browser, for example, Safari or Chrome.
- Hybrid applications (hybrid apps) - the user works with the web application shell, that is, interacts with the web content through the interface of the native application.
Detox
In our opinion, Detox is convenient for applications written in React Native. Tests are written in JavaScript, while iOS and Android applications are generated from the same JavaScript code and are as similar as possible. This allows you to use the same tests for both platforms.
The key feature of Detox is gray box testing. In this case, the framework has some access to internal mechanisms, which allows us to relate the external behavior of the application to what is happening at a deeper level.
Detox can access memory and monitor running processes. The principle of gray-box helps to deal with the instability, which is expressed in the fact that with end-to-end testing:
- The test can randomly crash even without changes in the code;
- Results are not deterministic - due to the large number of heterogeneous functionalities and processes within the application, the results of each launch can be unpredictably different from each other.
- Testers are forced to synchronize manually, which leads to a decrease in the reliability and quality of the results.
Oddly enough, the “gray box” shows not only better stability, but also a higher speed than the “black box”. Avoiding all sorts of pauses, waitUntil, gray-box can be 5-10 times faster.
Detox does not need WebDriver, working with the native driver via JSON. It uses native methods directly on the device. Inside this framework, apply EarlGrey for iOS and Espresso for Android.
The framework works with both emulators and physical devices.
Link to the documentation .
Appium
The advantage of Appium is that it is possible to write tests for each of the platforms using a single API, without resorting to converting the application to any particular framework compatible with the framework.
When testing frameworks from vendors are used - that is, you work with the original application. For Android 4.2+, respectively, UiAutomator / UiAutomator2 is used, and for iOS 9.3+ - XCUITest. WebDriver (also known as Selenium WebDriver) is used as a framework framework.
Principles of Appium:
- No need to recompile the application or modify it to automate testing.
- It is not necessary to be attached to one language or framework.
- It’s not necessary to reinvent the wheel when it comes to automation APIs.
Using Appium is justified when you need a tool to automate testing on multiple platforms at once. It is useful if you have specialists with experience in testing web applications, but no experience in automating the testing of mobile applications.
In general, it is a flexible tool that can be modified to the needs of the project without the need to adapt to a limited set of development languages.
Link to the documentation .
Ranorex
Paid comprehensive tool for testing desktop, mobile and web applications. Allows you to test as with the use of programming, and without the use of scripts. Provides the ability to test not only through emulators, but also on live devices.
The tool allows you to create and configure tests, as well as manage them centrally. You can create a test in the control center and run it in various external environments and on any devices.
It is easily integrated with the existing CI environment: with application management systems such as Jira and TFS, as well as with version control systems, such as Git and SVN.
In Ranorex, data-driven testing with data loading from SQL, CSV, Excel has been improved.
The tool is absolutely suitable for any device, supports parallel testing on each of them.
Combines all three approaches to testing: black-box, white-box and gray-box.
Link to the documentation .
Testcomplete
Paid environment for automating the testing of mobile, web and desktop applications. It supports Android and iOS, and in terms of application types: native, web-based applications and hybrid.
Focused mainly on functional and unit testing, the tool also provides the ability to perform many other types of testing:
- Regression;
- Data driven testing;
- Distributed testing and more.
In TestComplete there is a Recorder - in it tests are created by recording actions and setting up commands in the editor. Then they can be launched either directly in the tool itself or exported to third-party applications.
This tool recognizes objects and controls, offering special commands to emulate user interaction with them. Integrates with Jenkins, Git and Jira, which allows you to run continuous seamless testing.
Link to the documentation .
Summing up
Planning to test this or that mobile application, pay attention to the tools listed above. Each of them has its own characteristics, and sometimes limitations.
Consider an example. If you are faced with the task of testing a small application in a short time, first of all you need to take into account such factors as the type of application being tested and the experience of your specialists. If the developer writes tests, it is better to choose a native language and a tool for its platform (see the table below). If the tests are done by SDET specialists who are familiar with other languages ​​(Java, JavaScript, Python, etc.) and have worked with Selenium, it is convenient to use Appium. If there is no experienced SDET in the team, and QA specialists will write tests, it is better to choose paid frameworks, since they have utilities for recording tests and more stable technical support than in open source frameworks.
From our practice:
We worked with one online store, which had two mobile applications - on iOS and Android. To cover the main user scenarios with tests, we chose Appium for several reasons:
- cross-platform, partially reusable code
- suitable for end-to-end tests, can work with web
- the presence of a team of professionals who know Selenium well, which serves as a shell of this framework.
As a result, Appium fully met expectations, we successfully conducted tests for iOS and Android. It should be borne in mind that similar end-to-end tests with Appium are not carried out at each merge request, since this takes a lot of time.
In conclusion, we offer you a table that will help you choose a tool for your project. It is worth noting that in some cases the division in the table is conditional. Somewhere for simplicity, a generalization has been made and only the most basic parameters are given. Testing tools are constantly evolving, so when choosing a framework, it is important to check the actual documentation.
Thanks for attention!
Source: https://habr.com/ru/post/459292/
All Articles