Test automation for Android applications

Testing is an important part of the application development process. For Android, testing is especially important, since the devices are very different from each other:

• The size and resolution of the screen.
• Android version.
• Form factor
• Command system processor.
• Presence of front camera, NFC, external keyboard, etc.

Therefore, the application has to test on a variety of devices.
The testing process includes various types of testing. Consider the process of functional testing of the application manually. The tester installs the application on the device, thoughtfully checks all the functionality, then returns the device to its original state. And so for each application and each device. The obvious disadvantage of this method is a great investment of time for regular testing.
The obvious advantage of automated testing - it can be regularly carried out without special costs. For example, every night, test a fresh build of the application on the entire set of available devices, and in the morning analyze the results and correct errors.
This article will discuss the means of implementing automatic testing. Only tools included in the Android SDK or distributed under an Open Source license are considered.

The concept of automatic testing

The task is to automate with the greatest accuracy the actions that the tester performs. Let's consider them. There are several applications and several Android devices. For each application and each device, the following steps are performed:

1. Install the application on the device
2. Application launch
3. Testing the application in a selected way
4. Uninstall application
5. Reset device status

At each step, you need to collect and analyze data, such as logs and screenshots. Then on the basis of this data to form a test result.

Next, we consider the means to automate the listed steps.
')

Manage Android devices

First you need to select the computer on which the automatic testing will run and configure the Android SDK on it. Examples are given for a computer running Linux.

On all tested devices, you need to disable the lock screen and maximize the wait time. For some testing methods, you need to turn off the screen orientation change.

The Android SDK has two utilities for managing devices: adb and MonkeyRunner.

I will try to describe in detail the automation of the actions used in testing. For those familiar with ADB and MonkeyRunner, it makes sense to immediately go to the section " Methods of automated testing ."

Management using the ADB utility

ADB (Android Debug Bridge) is a utility for managing Android devices from the command line. Official ADB documentation: developer.android.com/tools/help/adb.html

The adb utility is located in the <android_sdk>/platform-tools/ directory. The path to this directory is recommended to be registered in the PATH environment variable.

ADB operation check

Install and configure the Android SDK, connect the Android device to the computer and execute the command:

 adb devices 

The command will list all connected devices. If the device list is not empty, then ADB is configured and working.

Work with multiple devices

To specify the ADB with which device you need to work, you must specify the serial number of the device after the -s key:

 adb -s <serial_number> <command> 

For example:

 adb -s <serial_number> logcat 

The serial number of the device can be viewed with the adb devices command. The -s allows you to work simultaneously with several connected devices. In the future, the -s in the commands, I will not specify.

ADB basic commands

Open console on device:

 adb shell 

Run the command on the device:

 adb shell <command> 

Android has many standard Linux utilities: ls, cat, dmesg, ...

Install the application from the apk file:

 adb install example.apk 

Remove application:

 adb uninstall <package> 

The package name can be obtained from the apk file with the command:

 aapt dump badging example.apk | grep "package" 

Download file from device to computer:

 adb pull <path-on-device> <file> 

Download file from computer to device:

 adb push <file> <path-on-device> 

Note:
Most directories on the device are read-only. Write access is allowed in the /sdcard directory (programs cannot be run from it) and /data/local/tmp/ .

Application launch:

 adb shell am start -n <package>/<activity> 

Starts the specified activity. The name of the activity that is started when the application is selected in the menu can be obtained from the apk file with the command:

 aapt dump badging example.apk | grep "launchable-activity" 

Reading logs

Reading logs on Android is done using the logcat utility.
Logcat utility home page: developer.android.com/tools/help/logcat.html

Read logs from the device (blocked until Ctrl-C is pressed):

 adb logcat 

Clear the log buffer on the device:

 adb logcat -c 

Read the log buffer on the device (gives the current contents of the buffer, is not blocked):

 adb logcat -d 

Example:

 adb logcat -c #    #   adb logcat -d > file.log #       file.log 

Taking screenshots using screencap utility

The screencap utility saves the current screen contents to a graphic file:

 adb shell screencap /sdcard/screen.png adb pull /sdcard/screen.png screen.png adb shell rm /sdcard/screen.png 

The screencap utility is available on phones with Android 4.x and higher. On previous versions of Android, screenshots can be taken using MonkeyRunner.

Sample BASH script for testing an application using ADB

Control with MonkeyRunner

MonkeyRunner provides an API for writing scripts that control Android devices. With the help of MonkeyRunner, you can write a Python script that installs an Android application, launches it, simulates user actions, takes screenshots and saves them to a computer. MonkeyRunner uses Jython to execute scripts.



MonkeyRunner utility homepage and API description: developer.android.com/tools/help/monkeyrunner_concepts.html

Reading logs with MonkeyRunner

File log.py:

 # coding: utf-8 from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice def log(fn, device): msg = device.shell('logcat -d') f_log = open(fn, 'at') if msg is None: msg = 'None' f_log.write(msg.encode('utf-8')) f_log.close() device.shell('logcat -c') if __name__ == '__main__': device = MonkeyRunner.waitForConnection() device.shell('logcat -c') #    # ... log('example.log', device) #   

Run:

 monkeyrunner log.py 

The script will write the logs to the file example.log in the current directory.

Taking screenshots

Screen.py file:

 # coding: utf-8 from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice if __name__ == '__main__': device = MonkeyRunner.waitForConnection() image = device.takeSnapshot() image.writeToFile('screenshot.png','png') 

Run:

 monkeyrunner screen.py 

The script takes a screenshot and saves it to the screenshot.png file in the current directory.

Device control example with MonkeyRunner

Run:

 monkeyrunner monkeyrunner_test.py 

Automated Testing Tools

Monkey testing

Imagine that the device got into tenacious paws of a very active and creative monkey - the monkey utility is designed to imitate a similar situation.

The monkey utility is included with the Android SDK. The utility sends pseudo-random user actions to the device. The command line parameters specify the number of user actions, the ratio of their types and the name of the package being tested, so that, for example, the monkey does not go beyond the limits of the application being tested and does not start sending SMS to all contacts from the address book.

Examples of use and a list of parameters are given on the home page: developer.android.com/tools/help/monkey.html

The main advantage of monkey is the lack of support costs. In addition, stress testing an application with a stream of arbitrary events can detect nontrivial errors.

Disadvantages of testing with monkey:

• Ineffective for testing functionality caused by a complex sequence of actions. For example, monkey will not be able to authenticate and the main functionality of the application will be ignored.
• Games with complex controls that require quick response and complex gestures will be completed at the very beginning, or will not start at all.
• Errors found with monkey are very difficult to reproduce.
• No check of application status.

Using the monkey utility, you can easily test any application - this is a good starting point. It is possible that this method will show adequate results for a particular application. If the quality of testing is unsatisfactory, then you should use other methods of testing.

MonkeyRunner Testing

Using scripts that use the MonkeyRunner API, you can not only develop a framework for a testing system, but also write scripts to test a specific application on a specific device.

Advantages:

• Flexibility - you can implement almost anything.

Disadvantages:

• The complexity of writing scripts, even in simple cases.

As a rule, this method is not justified - writing scripts takes a long time. However, in particular cases, this method may work.

Testing with getevent / sendevent

The getevent and sendevent utilities allow you to record a sequence of user actions, and then play this sequence. Utilities are located on the Android device itself and do not require root access.
Advantages:

• The sequence of actions can be recorded at no additional cost during manual testing, if it is already underway.
• Scripting does not require programming skills.

Disadvantages:

• The sequence of actions must be recorded separately for each application and for each device. When changing the application interface, all recorded actions must be performed anew.
• There is no application status check. For example, when testing a browser, a page opens. If it opens longer than at the moment of recording, then further actions will be performed before the page is fully loaded and the result will be incorrect. Sometimes it is possible to write a script in such a way that in all such cases the waiting exceeds the maximum possible.
• Fast and complex sequence of actions will be played longer than recorded - therefore, the method is not always suitable for testing dynamic games, where the reaction time and timeliness of the action are critical.

Record a sequence of actions:

 #    #    ,    Ctrl-C adb shell getevent -t > events.txt #      ./decode_events.py events.txt > events.sh #     adb push events.sh /data/local/tmp/ #     adb shell chmod 755 /data/local/tmp/events.sh #   adb shell sh /data/local/tmp/events.sh 

Recorded actions must be played on the device.

Robotium Testing

Unlike the previously discussed methods, Robotium is not included in the Android SDK, but distributed under an Open Source license.

Robotium home page: code.google.com/p/robotium

The main difference of Robotium is that test actions are described at the application interface level. In the previously discussed methods, test actions were explicitly or implicitly described at the input device level.

For example, in the application you need to click the "OK" button. Using the MonkeyRunner script, pressing the button is implemented as: “Touch the screen point with coordinates (x0, y0)”. Using Robotium, this is implemented as: “Press the button with the text“ OK ”.”

When actions are described at the application interface level, they can be made independent of the location of the interface elements, the screen resolution and the device position.

In addition, Robotium allows you to check the application's reaction to the action.

For example, after clicking on the “OK” button in the application, a list should appear with the element “Item 1”. With the help of Robotium you can check whether the list has appeared with such an element

If you perform checks after each action, it is easy to detect at which step the error occurred.

Disadvantages:

• For each application, it is necessary to develop a test script in the Java language. This requires programming skills and time-consuming.
• If you change the application interface, the test script will have to be modified.
• Writing a Robotium script is more difficult than writing actions using getevent / sendevent.

In general, Robotium allows you to develop the highest quality test scenarios at adequate cost.

Comparison of testing methods

Test method	Virtues	disadvantages
Monkey - a stream of random user actions.	There are no maintenance costs. Does not depend on the device. Stress testing can detect nontrivial errors.	The quality of testing varies from application to application. The errors found are difficult to reproduce. No check of application status.
MonkeyRunner is a device management script.	Flexibility.	The complexity of writing and maintaining scripts even for simple applications.
getevent / sendevent - record / play user actions.	Recording a sequence of actions does not require programming skills.	The recorded sequence of actions applies only to one device with a fixed orientation. When changing the application interface, you need to re-record the sequence of actions. No check of application status.
Robotium - a script for testing the application's interface with state checking.	Actions are described at the application interface level. The script can be independent of the screen resolution and device orientation. After completing the action, you can check the status of the application.	The complexity of writing scripts in the Java language. When the application interface changes, the script will have to be modified.

Results analysis

As a result of testing the application using the methods listed above, we received logs and screenshots. Now they need to be analyzed for errors.

Log Analysis

For starters, you can do a search by substrings:

• I / DEBUG
• FATAL EXCEPTION
• WIN DEATH

The list can be supplemented as errors are detected during manual testing.

Screenshot Analysis

In the process of testing manually, you can prepare a series of screenshots at key points of testing, and then compare them with the screen contents in the process of automated testing. This will determine if the automated testing process is proceeding correctly and detect errors.

It is also useful to compare the screenshot before and after running the application - this allows you to determine when an application crashes without messages on the screen and in the logs.

MonkeyRunner allows you to compare two screenshots with a given tolerance in percent:

 image1 = device.takeSnapshot() # ... image2 = device.takeSnapshot() if image2.sameAs(image1, 0.1): print 'image1 and image2 are the same (10%)' 

Unfortunately, the MonkeyImage API does not provide a function to load from a file. Therefore, to compare the saved screenshots, you will have to write your own function, for example, using the Python Imaging Library .

Reset device status after testing

After testing the application, the device must be returned to its original state.

This can be achieved in several ways:

• Repeatedly pressing the back button.
• Reboot the device.
• Restart zygote process.

Consider the first option, as the most appropriate.

Pressing the back button repeatedly

Press the "Back" button using MonkeyRunner:

 for i in xrange(0, 10): device.press('KEYCODE_BACK', MonkeyDevice.DOWN_AND_UP) time.sleep(0.5) 

In practice, this option is optimal, since it simulates the behavior of a real user.

Conclusion

In the note, some methods of automatic testing of Android applications, their advantages and disadvantages were considered. In addition, the tools included in the Android SDK or distributed under the Open Source license are reviewed.

It should be noted that automatic testing is not a panacea and does not replace other types of testing. A quality product is obtained with a well-constructed testing process that combines various methods.