Monkeyrunner. Pixel-perfect testing web pages on Android

Since Google released the monkeyrunner test automation tool, a lot of time has passed, and there has been no improvement in it. However, for the task of regularly checking web pages for correct layout, there was no better tool. Those who simply need a ready-made script for comparing screenshots of pages on an android with scrolling support can immediately download it from the link . Under the cut, it will be told what problems the mankayranner holds and how to overcome them.

Task

In short, the challenge was this: go through the pages of the list and make sure that they look the same as before (or almost the same). Needless to scroll through the pages to the end.

Algorithm of the decision

By itself, it looks quite simple:

1. Open page from list
2. Take a screenshot, compare it with the refrain
3. If the screenshot is not much different, scroll the page on one screen and repeat step 2
4. If the page has ended, or the result is very different from the original, go to the next page.

To take screenshots, monkeyrunner has a ready-made takeSnapshot function, and for comparing them there is an absolutely excellent ImageMagick , which allows you to evaluate how similar the images are, and not just require 100% match. ImageMedzhik has many comparison metrics, I use -metric RMSE.
To determine when it is time to finish scrolling the village down, it is enough to compare the last two screens, if they coincided, then the end is reached.
Python-junit-xml is used to export the results to CI or your favorite IDE.

Underwater rocks

Now the most interesting thing: if you do all this "in the forehead", then nothing will work. There are several reasons.
')
1. Windows
monkeyrunner has an interesting feature, monkeyrunner.bat under Windows in the process of execution replaces the current directory with the one where the mannei-manner itself lies, because it is so convenient for him. As a result, all relative paths and import directives stop working in our script.
To overcome this behavior, the script should determine its location by itself and continue to act only on absolute paths. This is done like this:

filepath = path.split(os.path.realpath(__file__))[0] BASE_PATH = path.split(filepath)[0].encode(FILENAMES_ENCODING) try: import config from junit_xmls import TestSuite, TestCase except ImportError: #dirty hack that loads 3rd party modules from script's dir not from working dir, which is always changed by windows monkeyrynner import imp config = imp.load_source('config', BASE_PATH+'/src/config.py') junit_xml = imp.load_source('junit_xml', BASE_PATH+'/src/junit_xml/__init__.py') TestSuite = junit_xml.TestSuite TestCase = junit_xml.TestCase 

2. Russian characters
The very first attempt to test the Russian Wikipedia failed, as the mankairnner is based on the second python and inherits all known problems of unicode and national symbols. I had to explicitly specify the encoding wherever possible, as well as slightly modify junit_xml, the author of which was unaware of the national symbols.
For example, to create files correctly, you need to transfer the name to unicode explicitly.

 filePath.decode("utf-8") 

In addition, on the Russian-language versions of Windows, the script will simply fall if the wrong path to ImageMagick is given, because Popen simply does not expect to receive Russian characters in the error message, and Windows localizes its messages.

3. Scrolling
If we scroll the same page 10 times in the same browser on the same device using MonkeyDevice.drag (), we get 10 different results. Drag simply does not guarantee that it will scroll the page equally. To solve this problem, I had to apply the following trick: comparing a new page with an old one, I cut several pixels off the top and bottom of the page and look for its place of its entry into the original (fortunately ImageMagick can even do this), then how much the page is lower or higher than the supposed position, and there is an error scrolled, you need to subtract it at the next scroll. Such feedback allows the script to not fall off on the third iteration and safely live to the end of the page.

4. Memory consumption
If you open 10-20 pages in a row, any browser will simply create 10-20 tabs, over time they will consume all the memory and the browser will simply stop responding normally to commands. At best, page loading will slow down, but usually this causes the monkeyrunner to simply fall off due to a timeout. To avoid this, I went for a small hack: before opening a new page, the current browser process is simply killed via adb, something like this

 device.shell('am force-stop ' + BROWSER_PACKAGE_NAME) 

This hack helps the standard AOSP browser quite well in the emulator, but it doesn’t really work on Chrome, Opera and FF, so in the end I just had to write my own wrapper on the web-view, in fact, a lightweight browser without tabs. Along the way, two other problems were resolved: the self-written browser does not ask for confirmation of access to the user's location and does not spoil the screenshots, and in addition it can be included in the script and it will be installed automatically via MonkeyDevice.installPackage ()

5. Failed connections
If you terminate the script and immediately restart it, most likely MonkeyRunner.waitForConnection () will simply fall down with an error, while the repeated call usually passes without problems. Therefore, the final script always tries to connect to the device twice.

6. Screen lock
If the device was blocked at the time of launch, the script will simply not work correctly. This can be avoided by simulating pressing the hardware menu button (for some unknown reason, it unlocks Android devices)

 MonkeyDevice.press("KEYCODE_MENU", MonkeyDevice.DOWN_AND_UP) 

But since this will lead to the appearance of a menu on an unlocked device, it is better to first make sure that the device is locked, you can do this via dumpsys

  lockScreenRegexp = re.compile('mShowingLockscreen=(true|false)') result = lockScreenRegexp.search(device.shell('dumpsys window policy')) if result: return (result.group(1) == 'true') raise RuntimeError("Couldn't determine screen lock state") 

Unlocking itself will not turn on the screen on the "sleeping" device, so before all these operations you should make a forced call to MonkeyRunner.wake ()
After the device is successfully unlocked, you can rely on the fact that other functions will start to work correctly.

Total

With all the edits, you can already check the correctness of displaying pages without fear of drops every 5 minutes, but for a better result, you should choose either a test browser or an AOSP Browser from the vanilla android.

P.S. The code is on the bitback and is open for copying and modifications.