The company in which I work develops custom software, including mobile applications based on Android and iOS. Due to the fact that the competition in this segment of the market is quite high, testers are not only responsible for meeting the final product specifications and customer expectations, but also put in a tight framework for budget and testing time. This encourages us to explore new tools and methods that would allow us to reduce testing costs and improve product quality.

Imagrium is the result of one of these studies. Technically, this is a Jython framework for cross-platform testing of mobile Android / iOS applications using image recognition , written by our company. It is presented in the form of a working PyDev project, which you can change to fit your needs. The code is distributed under the MIT license and is available on Github. In this article I will talk about the principles of the framework and its device.

Work principles

The framework is already about 2 years old, during this time it has grown and developed, incorporating the experience of using it on combat projects. In this case, the basic principles, perhaps, have not changed much. They are:

Use the same tests on different platforms
')
Usually, our customers order the application directly under several platforms, most often Android and iOS. It turns out that the specification is the same, the functional tests are the same, so the most effective from our point of view would be to have one test base for different platforms. In other words, the same test should take place under different platforms.

Separate resources from logic

Tools that allow cross-platform testing can be divided into two types:

• Providing a common API and mediation service that translates common calls to axis-specific calls (MonkeyTalk, Appium, Robotium).
• Using the pattern recognition method (Borland Silk Mobile, eggPlant).

We are big fans of the first approach, but at that time some tools were not yet there, the others were not very stable, and this forced us to try our hand at writing alternatives. At the same time, we did not want another record-and-replay instrument, because tests created by such instruments very quickly become very difficult to maintain. Why? Because in them resources are connected with the logic of work. For example, when changing a primitive operation, you will have to not only transfer pictures in all texts, but also change the necessary steps in the necessary tests. I wanted to avoid such meaningless and costly work using the popular PageObject pattern.

Maintain continuous integration and convenient debugging

All our projects are automatically built by Jenkins, so from the very first line of the code of this framework, we would like the tests for the application to be launched automatically too. Initially, it was not so easy because they decided to use Sikuli (as the most popular, documented and free solution) for working with images, which at that time did not have an easy way to separate the library from the IDE and only had support for Jython 2.5 (which, by the way, there was still no support for json, yes, yes), in unittest there were no heaps of tasty features (for example, auto-finding tests). However, over time, these difficulties were overcome, and now the test results are available in jUnit format, and Ant makes a beautiful page with statistics.

On the second point, if you saw the Sikuli IDE, then you understand that doing something a little more serious in it than a 10-line application is a pain. If only because there is no debugger. For us, this was enough to switch to using PyDev Eclipse, which is familiar to programmers and contains a lot of support features to speed development.

Ensure the same initial state of the system for all tests.

We borrowed the idea of ​​independence of tests from each other from jUnit because this makes it possible to run tests in parallel or perform random checks. Another task that pushed us towards this is the drop in tests in the middle of execution. We needed a system in which the fall of one test would not affect the performance of other tests. As a result, we decided to use snapshots (snapshots) of the emulator on Android and the reset function of the simulator on iOS.

Emulator downloads and responses should not critically delay test execution.

We really liked the speed of the iOS simulator, which could not be said about the boxed Android emulator. We had hopes for HAXM and x86 images supplied by Intel, but the trouble is that these images prior to version 4.4 did not contain the Google API, which is used in most of our applications (i.e., applications were simply not installed on these images). In turn, the image 4.4, which contains this API, was unstable for us (for example, it could crash when the application was reinstalled). Therefore, we chose Genymotion and VirtualBox to create and manage snapshots.

If you share these principles and think about some kind of framework for testing, we suggest you consider ours as one of the alternatives.

Environmental requirements

Imagrium works successfully on Java 7 x64 and Windows 7 x64 ( win branch of the repository) or MacOS 10.9 (branch of the ios repository).
Historically, only Android was tested under Windows, and only iOS under MacOS.

Demonstration of work

Before we take a quick look at the rules for writing tests on Imagrium, we want to show a video with the capabilities of the framework. This video shows the passing test for the HopHop app on iOS and Android.

How to write tests

The code written in Imagrium can be divided into two blocks: pages and tests . In this grouping, a test is a sequence of operations conducted on different pages, as well as transitions between pages. For example:

authPage = AuthPage.load(AppLauncher.box, self.settings) fbAuthPage = authPage.signUpFb() fbAuthPage.fillEmail(self.settings.get("Facebook", "email")) fbAuthPage.fillPassword(self.settings.get("Facebook", "password")) fbConfirmPage = fbAuthPage.login() lobbyPage = fbConfirmPage.confirm() 

As you can guess from the code, the test first loads the AuthPage page, then switches from it to fbAuthPage, fills in the necessary data and submits the form, confirms the user and sends it to the lobbyPage. In other words, the test goes through the pages and performs clear with t. operations tester, leaving the implementation of operations inside pages. Those. tests are a fairly simple group, and in order to learn how to write them, we need only learn how to write pages. It is also quite simple.

We write pages

The page is a Jython screen view of the / activity / application page. Technically, this is a class with fields and methods that govern these fields. In the most difficult case, it looks like this:

 class FbAuthPage(Page): email = ResourceLoader([Resource.fbEmailFieldiOS, Resource.fbEmailFieldiOS_ru]) password = ResourceLoader([Resource.fbPasswordFieldiOS, Resource.fbPasswordFieldiOS_ru]) actionLogin = ResourceLoader([Resource.fbLoginBtniOS, Resource.fbLoginBtniOS_ru]) def __init__(self, box, settings): super(FbAuthPage, self).__init__(box, settings) self.email = self.box self.password = self.box self.actionLogin = self.box self.settings = settings self.waitPageLoad() self.checkIfLoaded(['email', 'password']) def fillEmail(self, text): self.email.click() self.waitPageLoad() self.inputText(text) 

This snippet uses most of Imagrium's delicacies, so let's discuss this snippet with details in detail. framework capabilities.

Field Definition and Localization

Let's start with this line:

  email = ResourceLoader([Resource.fbEmailFieldiOS, Resource.fbEmailFieldiOS_ru]) 

This fragment links the email page field with a graphic resource (image or line). In this case, we associate the field with two resources at once (for the English and Russian locales). When a page asks for an email field, the system tries to find one of these images and returns the area associated with the first successfully found image.

According to our agreements, we store graphic resources in the res directory. To set a resource, we must pass to the ResourceLoader the path to the resource or text, for example:

  email = ResourceLoader("res/pages/ios/fb_auth/fbEmailFieldiOS.png", "Password") 

but for the flexibility and convenience of maintaining the code, we store the paths to resources in the variables of the src.core.r.Resource object.

Total : To manage a graphic item or line, you need to add a ResourceLoader declaration to the corresponding page.

Page initialization and field validation

The declaration of the field only sets the connection we need to use it, we need to refer to the field. Such calls occur either when using any operation on the field (for example, dragging or clicking), or when the page is initialized. The last operation is very important, because firstly it allows you to set the search area for the fields (we usually want to narrow it to the emulator boundaries) and check that we are on the page we want. Therefore, let's take a closer look at what happens in the page initialization code using the example of FbAuthPage .

First, we must always inherit the page from the src.core.page.Page class .

 class FbAuthPage(Page): 

as it gives us access to common methods of managing pages, for example, the method of waiting for full page load. We also need to start the parent constructor when the page is initialized.

  def __init__(self, box, settings): super(FbAuthPage, self).__init__(box, settings) 

and the last feature of initialization is the ability to set a search field for the field, usually it is the area occupied by the emulator, and to search only in it, we write

  self.email = self.box self.password = self.box 

The box parameter is initially considered after creating the emulator snapshot and before launching the application for the first time, and then it is passed from page to page. In more detail, two lines (vertical and horizontal on the emulator) from, for example, res / pages / android / hdpi / core , are taken , the system detects them on the page and builds an emulator area on them. If we do not assign fields to the fields, the system searches for the entire screen (which usually affects the quality of the search), although this is necessary for some exotic things, for example, to press a hardware button on the emulator.

Usually when we initialize a page, we check that we have the fields we need on it, and do it with this method:

 self.checkIfLoaded(['email', 'password']) 

Some fields may not be visible during initialization, specify only visible. If the page cannot find the fields, it sends an AssertionError exception that causes the test to fail and also reports the details to stdout.

Total : If you want the system to search for fields within the emulator, assign them self.box . Use page field validation with self.checkIfLoaded () .

What can you do with fields

In our FbAuthPage sample code , we used the click () method on the email field:

 self.email.click() 

Each field is actually represented by a Match object from Sikuli, so you can do everything with it that is written in the corresponding spec (drag, click, pinch, release, enter text, etc.).

Configuration access

Configuration is a very important part of running tests. In particular, it determines under which platform we want to run tests, which tests, for which application. Also in the configuration, you can register your variables and use them in tests or in the code pages.

In our example class FbAuthPage, you can see the line:

 self.settings = settings 

Here, the settings attribute is a sample ConfigParser associated with the current configuration file, so you can use all the methods from the official spec when working with it. Imagrium adds settings to each test, so you can use the configuration directly in the tests.

Example of working with the configuration:

 self.settings.get("Facebook", "email") 

OS dependent methods

For example, sometimes you have to press the back button (specific to Android) or enter text (in Sikuli, this is not possible to do simply because it enters the text asynchronously, which usually leads to the fact that some of the characters do not have time to add). To provide these features, classes were introduced that give your page OS-specific functionality.

In practice, it looks like this (multiple inheritance!):

 class FbAuthPageiOS(FbAuthPage, iOSPage): 

or so:

 class FbAuthPageAndroidHdpi(FbAuthPage, AndroidPage): 

Total: If we want OS-specific functions, we inherit from the required class.

Page organization

In the previous sections, we first talked about FbAuthPage , and then peredkachili on FbAuthPageiOS and FbAuthPageAndroidHdpi . In this section, we will discuss what these classes are, why they are needed, and how they are related to FbAuthPage .

Initially, we said that we want to run the same test on different platforms, but even within the same platform, we can make serious differences in the presentation of graphic resources. For example, resources under hdpi may differ from xhdpi, resources under iOS may differ from those under Android. At the same time , only resources differ , and the methods of working with them remain the same (or almost the same, adjusted for guidelines). It was necessary to come up with some solution to redefine resources for different platforms, and we used standard inheritance. In other words, our pages can be divided into two levels - general and platform-specific .

1. The general page contains the general logic of working with resources on the page. In our example, this is FbAuthPage . These methods do the same operations for both iOS and Android.
2. Platform-specific pages usually contain only resources that need to redefine the resources of the common page. They look something like this:
   

    class FbAuthPageAndroidHdpi(FbAuthPage, AndroidPage): email = ResourceLoader([Resource.fbEmailFieldAndroidHdpi, Resource.fbEmailFieldAndroidHdpi_ru]) 

   
   

With such a variety of pages, I would not want the test or the page to decide which pages to load. In Imagrium, it is the responsibility of the system to read the configuration and load the necessary pages when the page calls the load () method. In order for the system to load the correct classes, these classes must be specifically named. In details:

• iOS pages should be called [shared page] + “iOS” , for example: FbAuthPageiOS . Additionally, this page should be an FbAuthPage heir.
  
• Android pages have more parameters - this is the OS version and density. First, the system tries to load the class [common page] + "_" + "[major version]" + "_" + "[minor version]" + "_" + "Android" + "[density]" . For example: FbAuthPage_4_2_AndroidHdpi . If she did not find such a class, then she tries to load: [common page] + “Android” + “[density]” . For example: FbAuthPageAndroidHdpi .
  

If the class is still not found, the system returns an AssertionError with the appropriate description, which fails the test.

In practice, it looks like this:

In the test we write

 fbAuthPage = authPage.signUpFb() 

In the AuthPage general page there is a method:

 def signUpFb(self): self.actionAgreeTermsBtniOS.click() self.actionSignUpFb.click() return FbAuthPage.load(self.box, self.settings) 

This method calls the load () method, during which the system decides which page to call (for example, FbAuthPageiOS or FbAuthPageAndroidHdpi ).

Total : We implement the general logic of the page, and then, if another density / platform requires it, we add the modified resources to the appropriate classes of pages. The configuration system decides which page to trigger in the appropriate case.

How to run tests

It is assumed that you are cloning a project from Github, open PyDev in Eclipse, delete everything that is superfluous, add everything you need, and then want to test it. For testing, you need to install everything that is needed for the framework to work (see installation instructions), and then run from the project root:

 ant 

In a little more detail, this call runs run.py with a specific configuration file as the only input parameter (for example, conf / android_settings.conf ) as well as prescribing the necessary paths in the PATH and CLASSPATH variables, and then creates a page with test results.

In the general case, when testing, the emulator is started, the application is reinstalled on it, after which a snapshot of the emulator is created. Then, for each test, the same snapshot is launched, on which the next test passes.

Conclusion

We recommend using Imagrium when you need to test multiple platforms at once or when you do not have easy access to the application code to add locators for GUI elements. The most quickly learn the framework for those who programmed in Python, although the simplicity of the syntax of the language should facilitate the rapid learning to work with the tool. In this article, only the basics of working with Imagrium were given, I will explain in detail in subsequent articles if this article is interesting to the community, in detail about the configuration of the framework and its capabilities (for example, multi-user scripting). You can also read the official documentation on the Github project page and see this Hello, World example .