📜 ⬆️ ⬇️

Testing in Yandex. What we learned about the Appium framework, and whether it can be used for serious tasks

In the world of software testing, a very young direction is gaining momentum - automating the testing of mobile applications. And it is expected that the corresponding tools began to appear like mushrooms after the rain: Calabash , iOS Driver , Robotium , Selendroid , Appium . And it is about our experiments with the latest in mobile testing that I want to tell.

Recently, Appium is often mentioned at conferences and here, on Habré, there were already several posts about it. This is an open source framework written in JavaScript and designed to automate the testing of mobile applications. In essence, this is Selenium WebDriver, but for mobile applications. Appium allows you to manage Safari and Chrome on relevant devices, and therefore test websites under them, but an overview of these features and the nuances associated with them is a separate topic.

To protect you from those bumps that we ourselves filled, working with Appium, I want to tell you about what features of the framework we have encountered, what difficulties you may have and how to cope with them.

First, a little about why we chose Appium for ourselves. In Yandex, the automatic testing infrastructure is mainly designed to use the Java language, and I don’t want to build something new. The same Calabash, for example, is focusing on Ruby. Appium can be used with Selenium Grid out of the box. And we have a lot of applications, so I want a convenient test run in many threads. It is important to write them for all platforms using the same tool, because we have applications for both iOS and Android. In addition, Appium does not require additional modification of the application with which we deal in tests. Of great importance is the fact that there is a large community of third-party developers interested in the further development of this framework.

In addition, I want to highlight the main features of the tool:
')

But the choice of the tool is not limited to everything; the field of rakes begins with its direct use. So I will focus on these points in more detail.

Application Elements Inspector


The process of writing tests for a mobile application using Appium is very similar to writing tests for websites. You find any application elements (buttons, input fields, lists) by their selectors, interact with them and analyze their status. In the case of websites, we use various web inspectors to understand what element the selector has. In the case of mobile applications, each operating system offers its own solution. Appium also made a single application for all platforms, which allows you to look at the tree of elements and the current screenshot, as well as the values ​​of some attributes of the selected element.



But not everything is so rosy, and here's why:

Manage iOS Applications


The principle of operation comes down to converting your requests from the WebDriver API format into Apple's test framework commands - UI Automation . To run the tests on the simulator, you have enough of the app file of your application - no integration of third-party libraries is required. If you want to drive tests on a real device, you will have to be a little confused . In general, everything works out of the box: it is enough to install Appium from npm and install Xcode. But let's look at the pitfalls.

One test session - one application . While the test is being executed, it is impossible to exit the application specified at its launch without losing the session. If you want to go to Safari first, and then launch your application, then you will fail. Below I will talk about a possible solution to the problem.

On big data, Appium runs at low speed and Appium just dies on timeout . This causes an unpleasant problem. The way out of such a situation is competent thinking out test data. Avoid lists with a large number of items in the application under test.

Xcode does not allow you to run multiple simulators . So you can forget about parallel running multiple tests on one machine. There is only one legal way to brighten up the situation a bit. The server version of MacOS allows the user to create two virtual machines, each of which can work. The solution is expensive, since only 2 iOS simulators are obtained from one Mac mini, but we stopped there due to the lack of other options.

Port to control the browser - unchangeable . This item is about testing websites on the iOS simulator, which I promised not to touch, but its understanding explains better the impossibility of parallel test launches. The fact is that when trying to configure the launch of multiple emulators on a single iron machine, we did not stop at the variant with virtual machines. It turns out that if you have several accounts under one operating system, then each user will be able to run the simulator. But this is where the described problem occurs. Appium manages the browser of the simulator via Web Inspector, which in the case of the simulator listens to port 27753 on the local host. This port is not set, respectively, when starting from under different users, each of the simulators will try to occupy this port and nothing good will come of it.

You can not select the desired version of iOS on the simulator . When launched through the console, the simulator runs by default with the latest version of iOS, available in Xcode, and cannot be selected. Accordingly, in order to test the application on iOS 6 and iOS 7, you need to download Xcode 5 and Xcode 4 and switch to the correct version with the command sudo xcode-select -switch before running the tests.

The tool does not see transparent elements . In the Yandex.Maps mobile application, on top of the entire map lies a transparent element that catches the actions of your fingers. His eyes are not visible, but he is. And there are, for example, buttons for zooming in and out. And we see them with our eyes, but if you try using Appium to find the elements responsible for these buttons and try to tap them, then nothing happens. UI Automation begins to think that this element is not on the screen, and it is necessary to fix this by flushing the screen, but there is no place to scroll. As a result - a mistake. The solution to the problem is tap by code. Read the position and size of the required object. We add to the position of half the width and height and tapa. Works.

Unstable simulator in Xcode 5+ . Yes, problems are not always connected with Appium itself. Developers from large corporations also throw sticks in the wheels. So the iOS simulator, starting with Xcode version 5, began to fall with enviable regularity at startup. As a result, the Appium server has an argument responsible for the number of attempts to launch the simulator.

There is no way to prepare a simulator . Sometimes to test the application is not enough empty operating system. For example, when testing the Yandex.Disk application, we needed to check the download of unique files. I wanted to have a fresh image in the photo album when you run the test. If you run tests on a single machine, then there is no problem: you run the test, which in Safari downloads the picture to you, and then you run the test, which is already working with the application that uses the picture. But when working with Selenium GRID, different tests will end up on different simulators. To solve the problem, Appium had to be slightly modified. In JavaScript, a test was written that ran Safari, pulled a handle that returns the current timestamp as qr code, and saved the resulting image into a photo album. In turn, Appium began to monitor another parameter — DesiredCapabilities, and if it was true, then before returning the current session to the tests, our test for creating the file was run locally. Accordingly, when the main test began, there was already a picture in the albums. Uploading files is required not only in our tasks, soon it will be possible to do without crutches .

Manage Android Applications


With the Android operating system, the framework considered in this article is also pretty good in general. This time, WebDriver API calls are converted to calls from Google’s test framework - uiautomator . The application can be tested both on the simulator and on a real device. In some places it is even more convenient to work with Android than with iOS. For example, Android does not limit developer tests to a single application within a single session, you can log in and out of any applications an unlimited number of times per session. And running tests on a live phone will only require you to connect it to a computer, Appium does the rest for you. Now let's take a look at how Appium works with Android applications.

Difficulties

In the tree of elements there are only those that fall when the tree is built into the screen area. In the case of iOS, if an element is out of the screen's scope, then iOS will try to scroll through it itself, but Android doesn't know anything about such an element, because it is not in the current tree. Accordingly, if you need to tap on a button that you do not see right now, you will first have to skip it.

Images of Android under Intel, and under ARM. ARM is the most common, but the simulator has to emulate the processor architecture, and as a result, everything works very slowly. Intel is closer to reality (at least the processor architecture is no different from the one presented on the system running the simulator). On top of that, Intel itself launches the HAXM accelerator, which makes testing even more comfortable - everything works very fast. But there are situations when it turns out that the third-party library used in the application is compiled only under ARM. For such cases, you can run tests on the functionality that uses this library on slow ARMs, and the rest on fast Intel.

Unreal xpath. In the Appium inspector there is an automatically generated selector for the selected element in the form of xpath. But be careful: for Android it may be incorrect. The fact is that at the moment Appium does not distinguish // from /. He does not care how deep the descendant is in relation to the parent. And very often the wrong item is returned. The situation is promised to be fixed in Appium 1.0 . In the meantime, I can recommend using other types of selectors - by class or id - or try to check xpath, which offers you Appium.

If your application goes to the API via https, then your root certificate may not be on the list of trusted certificates. So after the next renewal of certificates on test servers, the application stopped working for us. Visually, it looks like the absence of the Internet, but you will not see any error messages. To fix the problem, you will have to manually pour the certificate into the simulator, and then import it in the settings. The system will ask you to set a PIN code for the phone. Put, Appium can unlock the phone.

OpenGL ES can slow down tests. Do not forget that in addition to acceleration in the form of HAXM, Android also uses graphics acceleration using OpenGL. So, locally, it will most likely work fine for you, and the tests will run relatively quickly, even on ARM. But on serious servers, where the video card is far from the most important component, OpenGL is likely to refuse to perform its task. Therefore, the tests will go slower than on the local computer. This must be considered.

In the tree of elements there may not even be those objects that are now visible on the screen. This is a very rare situation that we have encountered only once. In the Yandex.Music application it was not possible to find a point of contact with the search sadgest. How to get around the problem without modifying the application is not clear. Unless only interaction with the supposed coordinates of the elements will help.

Problems on the side of the Android SDK. If Apple in the iOS 7 simulator became all bad with stability, then Google has some failures in the work of the Android SDK itself. One recent example: in Android 4.2, erasing text has stopped working. Even the transfer of the backspace button code as a symbol did not help. As a result, all tests for renaming text were ordered to live long. In addition, some time ago it was impossible to get a screen shot, which is also unpleasant. So if something is not working for you, it’s not a fact that the problem is in Appium.

Distortion of the input audio signal. The simulator distorts the sound coming to the microphone, so your application will not be able to recognize the voice. Theoretically, you can make your patched image of the image of Android. So in the Bluestack simulator there are no problems with sound, but there is no uiautomator in it, so working with it through Appium will not work.

Useful features

Parallel test run. Unlike the iOS simulator, Android has great flexibility in terms of settings and can run an unlimited number of its instances on one machine. So you can tell the simulator the port on which adb will communicate with it. In addition, due to this, simulators will work separately, you will also know the simulator id, which is obtained from the emulator-{port} pattern. Then, when you start Appium, you will need to scatter servers on different ports and set a parameter pointing to another port for communicating with Appium with the simulator (using adb, only a small part of the commands are executed). As a result, the command to start the simulator will look something like this emulator-x86 -avd API18_X86_1 -port 5554 -sdcard 200MB-1.img -no-boot-anim -qemu -m 2047 -enable-kvm , and Appium itself will run line appium -a {server address} -p 4721 -U emulator-5554 -dp 6721 --full-reset --nodeconfig register-x86-port-4721.json . You will only have to change the ports for new copies of simulators and Appium servers. Register them all in Selenium GRID and get the infrastructure ready for parallel test runs.

The freedom of action. In the case of Android, you can do anything you like: exit the application specified when starting the test, go into another application, exit it and enter the third one. All this will be one session. Android allows you to work with all your system as a single application in a single session. So it will be much easier to set up a simulator from the test for your test. Save files, perform some actions in the settings of the system itself.

Proxy support. When can it come in handy? For example, to test the functionality of your application, which depends on the geographic region. By specifying a proxy from another country, you can beautifully check this functionality. This attribute in another situation was useful to us. Let's say you have a production server for the application and testing. But your developers did not take into account the possibility of switching the environment in the application, and you urgently needed to test the application on the test API. This is where the proxy will help. We raise nginx, we just write a perl redirector that will replace the address of the combat API with the one being tested, and force the simulator to go through our proxy with the redirector. So - without modifying the application - you can check the correctness of working with the API on various environments.

Interaction between the test and the Appium server based on WebDriver API


There can be only one problem: if it turns out that the method you need is not yet implemented. You can view the missing methods in the routing file. The rest of the use of WebDriver API is a huge plus Appium. After all, this allows you to use various third-party solutions that were intended for testing websites, for testing mobile applications. So here I will list only the benefits of using Appium.

Selenium GRID support . You lift the Selenium GRID server, put Appium servers on machines, create json configuration files , launch Appium with them. Everything. The infrastructure for parallel test run is ready.

No need to connect extra libraries . A classic artifact (if we are talking about Java) selenium-java to help you. For other languages, it is also sufficient to connect a library containing the WebDriver client.

Ability to use various add-ons for WebDriver . For example, our guys have developed an excellent library HtmlElements , designed for convenient storage of selectors of elements and the organization of the beautiful architecture of your tests like the PageObject template. This approach allows the use of practices already tested on testing websites in testing mobile applications.

Conclusion


What do we have in the end? Appium is still a raw but fast-growing product. The topic of automating the testing of mobile applications is still gaining momentum, so Appium is more likely an old Selenium than a modern WebDriver. The server may crash at the wrong moment, the test session may hang in Selenium GRID (after our pull request, since version 0.17.7, the error has been fixed). Documentation is not always relevant and sometimes after the next update that some attribute has suddenly become mandatory, you will learn only after studying the source code of the product. Errors are not corrected as quickly as we would like. For example, Appium for a long time did not work under ubuntu. The error was corrected quite recently. But, as I said earlier, the product is developing rapidly, he (at the time of writing the post) already has almost 1000 forks and almost 1200 likes on a githaba. It can be seen that the developers follow the wishes of users and take them into account. If earlier almost all development was carried out by several people from Sauce Labs, now, thanks to open source, third-party developers are involved in the emergence of new features and bug fixes. At the moment, we are considering using Appium as an experiment, but it is becoming more and more like a serious solution to practical problems.

PS While this post was being prepared, new versions of Appium: 1.0 and 1.1 were released. While we are just looking at them and not in a hurry to update. On the one hand, the new versions have fixed some of the problems described here. For example, "earned the correct xpath." But in version 1.0 it “earned” so much that, when requesting an “image”, it could return an “input field”. 1.1 fixed this problem. In general, not everything is still perfect, and if they corrected something in one place, it is not a fact that they did not break it in other places. We are waiting for stability.

Source: https://habr.com/ru/post/227571/


All Articles