Automated testing of mobile applications
I conducted a real study of the situation with automated testing of the interface of mobile applications. This is not about testing modules, but about testing the interface of the final application. And, yes, right on the phone!
Why do you need it? First of all, to guarantee the improvement of the quality of your software and improve the mood of testers.
What is the idea? Most often, testing of mobile applications is carried out by people: the tester takes the application, iPhone 3, iPhone 4, iPad, if he is unlucky, then he takes a couple of Androids and GalaxyTab, and tests your application, 80% of the tests are about the following scenarios:
- run the application, make sure that it does not fall;
- go to the place tab, make sure that all items are in place;
- go to one of the items to make sure that the description is in place;
...
')
Such tests are carried out after each release and take a very long time.
At one time, Selenium came to the rescue on the web, which allowed us to record the actions of the tester through a special browser plugin (do everyone remember the macros in MS Word?) And then play them automatically with a check of the result. You can run tests even on different browsers! We used this solution in our company, and it really works. Efforts to develop tests paid off.
Compared to the web, mobile development is still a very young field, and I did not expect to see good solutions for automated testing of interfaces. It turned out that they are more than enough. I want to tell you about some of them.
I'll start with Apple's standard solution, which is included in Instruments.
UIAutomation allows you to run test scripts written in JavaScript both in the emulator and on the device. You can test any of your applications, including production assemblies. Compile additional libraries is not required. You can run scripts from the console.
Apple has developed a special JavaScript API for testing. It describes all the standard interface elements. Elements are represented by a tree, there are special functions for quick selection of elements by type (buttons, tableViews, textFields ...). If you do not specifically specify the Accessibility Label in the program code, then the elements can be selected only by number.
Select the second cell in the first table.
I wrote a test script for our Goozy application, it logs in, checks that the “All” tab is selected, goes to the “Settings” tab, and logs out.
It took me an hour to write this script, at the same time I kept XCode open next to it and looked at how the elements are represented in the designer, how they are generated dynamically. In addition, scripts are written in a small window (on the screenshot UIAutomation is shown on the right), there is no syntax check, there is no support for version control systems.
[Illuminated code: paste.ly/7mnG ]
The bottom line: the platform is powerful and correct, but testers are unlikely to be able to write tests on their own, as the internal structure of the interface changes, tests will have to be rewritten.
UPD: In Xcode 4.3, it is possible to write tests. Running from the console is almost impossible.
UPD2: When working with UIAutomation, it is strongly recommended to set all Accessibility Label components
Record & Play solution, it is interesting that the tests are recorded and edited (!) Directly from the application under test on the phone or the emulator.
However, the phone must be connected to the computer to save the tests. And of course, in the application you need to compile several additional libraries.
Fonemonkey can save tests in UIAutomation format, so you can run them on “clean” applications after recording.
In my opinion, this is the only solution that testers can use on their own. On one condition, the tests will be run either by the developers, or they will automatically fall into the Continious Integration system.
UPD: Export to UIAutomation works very crookedly, I could not run the resulting scripts.
Squish is a solution for testing Windows, Mac, Qt and iOs applications. The license for a specific developer is (named license) $ 2400.
The system integrates into HP Quality Center, Seapine TCM or provides an Eclipse based IDE. You can test applications on an emulator or device, record tests are available (record & play). Scripts can be in Javascript, Python, Perl or Tcl.
The system has two drawbacks: to work on the device (even for running the tests) you need to compile a separate version with their library; the second minus is the eclipse, if you don’t go bankrupt on HPQC, it’s unlikely that testers can fully work with the eclipse-based IDE.
In general, I do not see any fundamental differences from the free FoneMonkey. Unless Squish supports multitouch (support for automatic gestures).
One of the most promising platforms. JamoSolution allows you to test iPhone, Android, Windows Mobile and BlackBerry applications. At the same time for all platforms record test (record & play) is supported and you can test iPhone applications on Windows!
They have great demo videos, here is the iPhone testing:
Under Android:
Very pleased to see the phone screen in the IDE. JamoSolution works through a special agent on the phone, that is, your applications will remain unmodified. The phone can be connected by wire or via WiFi.
The most popular solution is Robotium . “It's like Selenium, but for Android,” say the developers, but this is a bit wrong. Robotium does not support the recording of tests (Record & Play) and does not know how to take screenshots.
Tests are written in Java, in fact, Robotium is a library for ordinary Unit tests, there is no possibility to run tests on the device (remote control). To test applications, you need to build them with this library. It is possible to test applications without source, but the process is not trivial.
While I was conducting research, a new solution for Android appeared. TestDroid is a Eclipse plugin that allows you to record tests (record & play) in Robotium format. The system interacts with the phone through the standard Android debugger.
One license for TestDroid Recorder costs $ 600 per year. The developers promise to release another TestDroid server, it will allow you to create clusters for testing. Connect several Android phones to the server and run tests in parallel and remotely.
Unfortunately, the solution is still in beta testing and does not generate tests on my poppy.
In addition, judging by the logs, not all actions are recorded in the tests. The solution is promising, but wait a bit.
At the time of this writing, there were no utilities for automating the testing of the WP7 interface.
Surely you are familiar with the concept of " Black Box ", and so, there are at least two mobile testing systems working on this principle.
This is Perfecto Mobile and Device Anywhere .
The essence of these decisions is that they have a farm of real mobile phones connected via cable and a webcam transmitting images from the phone. It looks like this:
The picture from the webcam is inserted into the photo of the phone. Fully controlled by mouse.
The core business of Perfecto Mobile and Device Anywhere is not even automated testing, but selling time-consuming devices. They have hundreds of different phones (iPhone / Android / Blackberry ...) and an hour of working with one phone costs about $ 15.
This is a very convenient solution if you are connected with many platforms and especially with Android, for which there are dozens of different devices.
But, unfortunately, when I tested both systems from Russia, everything was terribly slow, a big lag when interacting with the phone. Testers would be very nervous in this mode.
Very interesting implemented automatic testing .
Checking the results is performed by checking the pictures and recognizing the text on the OCR screen . One of the platforms uses the famous ABBYY :-)
By the way, testing is available only for Enterprise clients and is discussed individually each time, so I failed to try it.
Why do you need it? First of all, to guarantee the improvement of the quality of your software and improve the mood of testers.
What is the idea? Most often, testing of mobile applications is carried out by people: the tester takes the application, iPhone 3, iPhone 4, iPad, if he is unlucky, then he takes a couple of Androids and GalaxyTab, and tests your application, 80% of the tests are about the following scenarios:
- run the application, make sure that it does not fall;
- go to the place tab, make sure that all items are in place;
- go to one of the items to make sure that the description is in place;
...
')
Such tests are carried out after each release and take a very long time.
At one time, Selenium came to the rescue on the web, which allowed us to record the actions of the tester through a special browser plugin (do everyone remember the macros in MS Word?) And then play them automatically with a check of the result. You can run tests even on different browsers! We used this solution in our company, and it really works. Efforts to develop tests paid off.
Compared to the web, mobile development is still a very young field, and I did not expect to see good solutions for automated testing of interfaces. It turned out that they are more than enough. I want to tell you about some of them.
iphone
UIAutomation
I'll start with Apple's standard solution, which is included in Instruments.
UIAutomation allows you to run test scripts written in JavaScript both in the emulator and on the device. You can test any of your applications, including production assemblies. Compile additional libraries is not required. You can run scripts from the console.
Apple has developed a special JavaScript API for testing. It describes all the standard interface elements. Elements are represented by a tree, there are special functions for quick selection of elements by type (buttons, tableViews, textFields ...). If you do not specifically specify the Accessibility Label in the program code, then the elements can be selected only by number.
window.tableViews () [0] .visibleCells () [1] .tap ()
Select the second cell in the first table.
I wrote a test script for our Goozy application, it logs in, checks that the “All” tab is selected, goes to the “Settings” tab, and logs out.
It took me an hour to write this script, at the same time I kept XCode open next to it and looked at how the elements are represented in the designer, how they are generated dynamically. In addition, scripts are written in a small window (on the screenshot UIAutomation is shown on the right), there is no syntax check, there is no support for version control systems.
[Illuminated code: paste.ly/7mnG ]
The bottom line: the platform is powerful and correct, but testers are unlikely to be able to write tests on their own, as the internal structure of the interface changes, tests will have to be rewritten.
UPD: In Xcode 4.3, it is possible to write tests. Running from the console is almost impossible.
UPD2: When working with UIAutomation, it is strongly recommended to set all Accessibility Label components
Fonemonkey
Record & Play solution, it is interesting that the tests are recorded and edited (!) Directly from the application under test on the phone or the emulator.
However, the phone must be connected to the computer to save the tests. And of course, in the application you need to compile several additional libraries.
Fonemonkey can save tests in UIAutomation format, so you can run them on “clean” applications after recording.
In my opinion, this is the only solution that testers can use on their own. On one condition, the tests will be run either by the developers, or they will automatically fall into the Continious Integration system.
UPD: Export to UIAutomation works very crookedly, I could not run the resulting scripts.
Paid fellow
Squish
Squish is a solution for testing Windows, Mac, Qt and iOs applications. The license for a specific developer is (named license) $ 2400.
The system integrates into HP Quality Center, Seapine TCM or provides an Eclipse based IDE. You can test applications on an emulator or device, record tests are available (record & play). Scripts can be in Javascript, Python, Perl or Tcl.
The system has two drawbacks: to work on the device (even for running the tests) you need to compile a separate version with their library; the second minus is the eclipse, if you don’t go bankrupt on HPQC, it’s unlikely that testers can fully work with the eclipse-based IDE.
In general, I do not see any fundamental differences from the free FoneMonkey. Unless Squish supports multitouch (support for automatic gestures).
Jamosolution
One of the most promising platforms. JamoSolution allows you to test iPhone, Android, Windows Mobile and BlackBerry applications. At the same time for all platforms record test (record & play) is supported and you can test iPhone applications on Windows!
They have great demo videos, here is the iPhone testing:
Under Android:
Very pleased to see the phone screen in the IDE. JamoSolution works through a special agent on the phone, that is, your applications will remain unmodified. The phone can be connected by wire or via WiFi.
Android
Robotium
The most popular solution is Robotium . “It's like Selenium, but for Android,” say the developers, but this is a bit wrong. Robotium does not support the recording of tests (Record & Play) and does not know how to take screenshots.
Tests are written in Java, in fact, Robotium is a library for ordinary Unit tests, there is no possibility to run tests on the device (remote control). To test applications, you need to build them with this library. It is possible to test applications without source, but the process is not trivial.
Testdroid
While I was conducting research, a new solution for Android appeared. TestDroid is a Eclipse plugin that allows you to record tests (record & play) in Robotium format. The system interacts with the phone through the standard Android debugger.
One license for TestDroid Recorder costs $ 600 per year. The developers promise to release another TestDroid server, it will allow you to create clusters for testing. Connect several Android phones to the server and run tests in parallel and remotely.
Unfortunately, the solution is still in beta testing and does not generate tests on my poppy.
In addition, judging by the logs, not all actions are recorded in the tests. The solution is promising, but wait a bit.
Windows Phone 7
At the time of this writing, there were no utilities for automating the testing of the WP7 interface.
Absolute Black Box
Surely you are familiar with the concept of " Black Box ", and so, there are at least two mobile testing systems working on this principle.
This is Perfecto Mobile and Device Anywhere .
The essence of these decisions is that they have a farm of real mobile phones connected via cable and a webcam transmitting images from the phone. It looks like this:
The picture from the webcam is inserted into the photo of the phone. Fully controlled by mouse.
The core business of Perfecto Mobile and Device Anywhere is not even automated testing, but selling time-consuming devices. They have hundreds of different phones (iPhone / Android / Blackberry ...) and an hour of working with one phone costs about $ 15.
This is a very convenient solution if you are connected with many platforms and especially with Android, for which there are dozens of different devices.
But, unfortunately, when I tested both systems from Russia, everything was terribly slow, a big lag when interacting with the phone. Testers would be very nervous in this mode.
Where are the automatics?
Very interesting implemented automatic testing .
Checking the results is performed by checking the pictures and recognizing the text on the OCR screen . One of the platforms uses the famous ABBYY :-)
By the way, testing is available only for Enterprise clients and is discussed individually each time, so I failed to try it.
pros
- Black box - almost no interference with your application;
- One tool and test script for all mobile platforms;
- The variety of devices;
- You can test any applications: native, html5, mobile sites.
Minuses
- automated testing is expensive;
- big lag when interacting with the phone.
- What platform do you develop?
- What kind of application? (games, banking, business applications, medicine)
- Do unit tests why?
- Automate UI testing, why?
- If you were to automate, which tool would be ideal for you?
Source: https://habr.com/ru/post/123026/
All Articles