Hi, Habr! Today I want to tell you how we built from scratch a flexible and extensible system for performing autotests on Android smartphones. Now we have about 60 devices used for regression testing of Mail.Ru Mail mobile application. On average, they test about 20 application builds daily. About 600 UI tests and more than 3,500 unit tests are performed for each build.

Self-tests are available around the clock - they save a lot of time testers and allow us to produce a quality application. Without them, we would test each build for 36 hours (including waiting time) or about 13 hours without waiting. Together with the assembly, translation actualization, with a workload of agents with autotests, testing takes an average of 1.5 hours, which allows us to save testers work weeks every day.
')
We will consider how to do everything from the very beginning to those who are involved in writing autotests, not infrastructure: from buying a phone, flashing it, and ending with creating docker containers within which the phone will be accessible for autotests.

What phones to choose for autotests on Android?

When Android was just becoming popular, test developers had a choice of two evils: to buy expensive zoo phones or work with slow and buggy virtual machines. Today, everything is somewhat simpler, cheaper “economy” -class devices have appeared on the market, and virtual Android has acquired an image for x86 and HAXM. However, the choice still remains, many prefer virtual machines for autotests, but telephones have already become an affordable option even for a modest autotesting budget. So how do you choose a phone for regression autotests and what equipment do you need for it to work 24/7 all together?

The handset market is very big - your eyes run up. What are the criteria to set when choosing a phone? After a series of trials and errors, I got such a list (I’m lowering the price for the device, with it, I hope everything is clear):

1. It is possible to get root.
2. It is possible to unlock the boot section of the phone.
3. The phone has a version of Android, as close as possible to stock, or similar ones can be installed (so that you don’t have to shovel a bunch of test options for different interfaces).
4. The RAM on the phone should preferably be from 1 GB in size (you can work on a smaller one, but even if the tests are written stably, various checks on the display of “heavy” objects on the phone with low RAM will be long).
5. It's great if the phone has a long support from the manufacturer / users, then we still have a chance to extend his life with new versions of the OS.

The main part of the criteria is quite transparent. If it turns out that something is wrong on the phone, then we should at least have a chance to make it work ourselves. :) Unfortunately, most of the criteria - these are not the things that you can ask the seller when buying, so first of all our way lies on forum.xda-developers.com and w3bsit3-dns.com.ru/forum, where you can learn about the market model all the details. In addition to all the listed criteria, if the model is already selling for a long time, pay attention to the feedback on the marriage and the time of memory and battery life on the forums, without them your device will turn into a pumpkin. Problems of the screen, buttons, casing, speakers and other things that usually interest the user, as a rule, should not scare you: the phone will work fine with the marriage of these elements, although everything depends on the specifics of the project.

After choosing a phone model, it is best to order one or two pieces for preliminary tests, check that the operating system does not prepare any surprises, all the written tests for them pass correctly and that the hardware corresponds to the specifications declared by the manufacturer. In my practice, the most severe punctures for buying were the following:

1. The phone model has a bunch of regional subspecies, while only part of them can be used to get the root or unlock the bootloader. Not only stumble on a Russian certified phone in an informal store is difficult - gray and white phones look the same - so many vendors or their suppliers rewrite model names, hardware specifications, regions, and even operating system versions. It is quite possible that inside the “Settings” in Android you see one model, inside the bootloader another, and in the shell, when you type getprop and get IDs, the third one. Just the phone went through a couple of hands and a couple of regions before you. At first, his owner was Verizon's user from South Dakota, then he passed it, in refurbished-state, the device somehow hit the merchant in Tel Aviv, who crookedly asked for his version of the operating system, and after some time the phone was bought by the seller in Moscow, which has already started selling it as a new one. The courier brings it to you, you take into your hands your new eight-core reflash Russian device, not suspecting that this is a six-core locked regional exclusive for contract users of a cellular operator from the United States.


^{The elements of the box and the phone with a "modern" filling and a high price, which, according to internal characteristics, turned out to be a flashing younger model from AT & T}

2. The same serial number. Here it is easier to identify the problem, but, unfortunately, even the official sellers are suffering from this, the lack of a serial number is the attack of many budget devices. If you are using adb while your autotest is working, and several devices are connected to the machine, you must either rewrite the adb code (it will see only one device), or, if the purchase was made according to the criteria above, enter the unique serial itself.


^{Typical values ​​of serials for budget phones}

3. Pseudo-random MAC address on the Wi-Fi module after rebooting the phone. It was a rather serious problem, because we learned about it when I was convinced that “everything is fine”, the phone suits us, and we ordered 20 of the same ones. :) In the process of autotest, phones sometimes reboot, after some time the tests began to fall due to the lack of access to the network via Wi-Fi, although everything looked normal: the connection to the network was after Wi-Fi was turned on / off module everything worked correctly. Just after reboots, at some point, a pair of phones turned out to have the same MAC, only the last one who let the Wi-Fi access point was allowed. On those phones, where the MAC address was eventually generated, I, unfortunately, did not find it, I had to place a script in the boot section that installed it forcibly on a unique one.


^{The phone shows the wonders of spoofing out of the box.}

However, if you follow the criteria listed above when choosing a phone, these problems should not be fatal - you can fix it all yourself and make the phone work as it should.

In addition to phones, to run autotests you will need a computer and USB hubs, here, too, there are several nuances. Constantly working phones need good power (at least 0.5 A per device, preferably more), many hubs on the market come with weak adapters and are not designed to have a constantly working phone connected to each port. With tablets it is even more difficult, nine-inch tablets with constant work are discharged, the screen is too big, you have to choose from seven-inch. We learned from practice that 6–7 phones can be connected to an adapter in 4 A (depending on their workload), i.e. most of the multi-port hubs with the “3 A adapter, 20 USB ports” characteristics, to put it mildly, useless. The coolest are server solutions, but their price is off the scale, so we limit ourselves to the user market. So that the phones are not discharged, it is worth taking four ports with 3 A power hubs, or six ports with 6 A power ports. If there are hubs with good power, but with a large number of ports, you can simply not use some ports.

Preparing the phone for work

For example, let's take one phone model, solve one of the problems of its operating system, and then try to assemble these devices into a primitive test bench for autotests. The phone itself is cheap and good, but with some flaws (described above). In particular, these phones have the same iSerial, adb sees only one device. :( We will not change it everywhere on the phone, but we will make adb separate phones different.

To do this, we will need to reflash the boot partition on the phone and install a custom recovery partition on the device so that you can protect yourself from unsuccessful experiments. Our phones have MT6580, i.e. Mediatek processor, which means that you can use SP Flash Tool for flashing. We also need a ready recovery.img image and a scatter file of the device. For almost all devices, they can be found on the Internet, on the same XDA and 4PDA resources, but if you wish, recovery can be recompiled for your device, using TWRP as the basis, and you can create the scatter file yourself. In any case, take our finished files and reflash:



After installing the recovery partition, save the backup partition through it and move it to your machine, usually the OS configuration files are located in this section. To hardcode your iSerial, you need to unpack the image of the boot partition of the phone, this can be done using Android Image Kitchen . Run unpackimg.sh and get the unpacked image in the ramdisk folder:



There are many init files that contain various variables, including the serial number.



Find the file where the serial number ${ro.serialno} , and replace it with your own number, for example 999222333019:

 find ramdisk/ -maxdepth 1 -name "init.mt*" -exec sed -i 's/${ro.serialno}/999222333019/g' {} +; 

We pack the image back using repackimg.sh, transfer it to the phone and install it using custom recovery. Now adb will differentiate the devices, we just have to turn on the developer mode on the phone and enable debug in the developer menu. Any similar problems can be solved in exactly the same way, almost everything in the phone can be reflashed or corrected if the testing tasks require it.

Test machine setup

As the host of our booth, we will use a regular desktop with Ubuntu installed on it. The phones can then be divided into separate virtual machines, used together or, as I do, divided into separate docker containers.

When ordering / assembling the machine to which the phones will be connected, there is a specificity. In addition to standard HDD / RAM / CPU, you need to pay attention to the number of USB controllers on the motherboard and the supported USB protocol. Phones running on USB 3.0 (xHCI) can significantly limit the maximum number of devices on the machine (usually 8 per controller, eventually 16 devices per machine with two controllers), so you should check whether it is possible to disable it and use only EHCI. Such options are in the BIOS or OS, it is best to forcibly disable xHCI in the BIOS, if you do not need high-speed devices.

Create a container for your phone.

If it is necessary for the slave / integration system agent to work with a separate phone, then they should somehow be separated. Our agents run in separate docker containers, each of which is accessible by one device, so we can distribute tasks in CI to separate devices, or rather, their capabilities (for example, a container with a tablet can perform tests that require a wide diagonal screen, and the container with the phone - tests, which need the ability to receive SMS-messages). An example of installing and configuring a system on Ubuntu is described below. Install Docker itself:

 sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D echo 'deb https://apt.dockerproject.org/repo < ubuntu> main' >> /etc/apt/sources.list.d/docker.list sudo apt-get update sudo apt-get install docker-engine 

We will use overlayFS as the driver (it works faster than the default one):

 echo 'DOCKER_OPTS="-s overlay"' >> /etc/default/docker 

Create a dockerfile, from which we will make images. Add the Android SDK to it:

 FROM ubuntu:trusty ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update -y && \ apt-get install -y software-properties-common && \ add-apt-repository ppa:webupd8team/java -y && \ apt-get update -y && \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-get install -y oracle-java8-installer && \ apt-get remove software-properties-common -y && \ apt-get autoremove -y && \ apt-get clean ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV ANT_VERSION 1.9.4 RUN cd && \ wget -q http://archive.apache.org/dist/ant/binaries/apache-ant-${ANT_VERSION}-bin.tar.gz && \ tar -xzf apache-ant-${ANT_VERSION}-bin.tar.gz && \ mv apache-ant-${ANT_VERSION} /opt/ant && \ rm apache-ant-${ANT_VERSION}-bin.tar.gz ENV ANT_HOME /opt/ant ENV PATH ${PATH}:/opt/ant/bin ENV ANDROID_SDK_VERSION r24.4.1 ENV ANDROID_BUILD_TOOLS_VERSION 23.0.3 RUN dpkg --add-architecture i386 && \ apt-get update -y && \ apt-get install -y libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 && \ rm -rf /var/lib/apt/lists/* && \ apt-get autoremove -y && \ apt-get clean ENV ANDROID_SDK_FILENAME android-sdk_${ANDROID_SDK_VERSION}-linux.tgz ENV ANDROID_SDK_URL http://dl.google.com/android/${ANDROID_SDK_FILENAME} ENV ANDROID_API_LEVELS android-15,android-16,android-17,android-18,android-19,android-20,android-21,android-22,android-23 ENV ANDROID_HOME /opt/android-sdk-linux ENV PATH ${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools RUN cd /opt && \ wget -q ${ANDROID_SDK_URL} && \ tar -xzf ${ANDROID_SDK_FILENAME} && \ rm ${ANDROID_SDK_FILENAME} && \ echo y | android update sdk --no-ui -a --filter tools,platform-tools,${ANDROID_API_LEVELS},build-tools-${ANDROID_BUILD_TOOLS_VERSION} ###   ,     ,  Bamboo  . .,    ,     ADD moyagent.sh /agentCI/ 

You can also add all the necessary libraries and files that the integration system agent will use to the pre-file. Build the dockerfile:

 docker build . 

Now we have an image with the Android SDK, it remains to make sure that it sees only one device. To do this, we will hook it through a symlink with udev:

 echo '"SUBSYSTEM=="usb", ATTRS{serial}=="$DEVICE_SERIAL", SYMLINK+="androidDevice1"' >> /etc/udev/rules.d/90-usb-symlink-phones.rules 

Instead of $DEVICE_SERIAL enter our $DEVICE_SERIAL serial. Restart the device rule definition:

 udevadm control --reload udevadm trigger 

Now in the path / dev / androidDevice1 we will have a symlink to the device, it remains to transfer it to the container at launch:

 docker run -i -t --rm --device=/dev/androidDevice1:/dev/bus/usb/001/1 android-docker-image:latest adb devices 

As soon as we make sure that only one phone is visible from the container, the integration system agent located in the container can be launched:

 docker run -i -t --rm --device= /dev/androidDevice1:/dev/bus/usb/001/1 android-docker-image:latest /bin/sh /agentCI/moyagent.sh 

By the way, the --device key began to work with symlinks relatively recently (master branch on Github ), before that it was necessary to generate from the realpath symlink using a script and already transfer it to Docker, so if you don’t have a device connection, add to udev in the parameter RUN+= such a script:

 realpath /dev/androidDevice1 | xargs -I linkpath link linkpath /dev/bus/usb/010/1 

After that, in older versions of Docker, you can add a phone like this:

 docker run --privileged -v /dev/bus/usb/010/:/dev/bus/usb/100/ -i -t android-docker-image:latest adb devices 

Everything, you can connect your slave to the integration system and work with it.

Conclusion

Physical mobile devices in the integration system sooner or later appear in any more or less large-scale project on Android - inevitably there is a need to cover errors, non-standard test cases or just features that require a real device. In addition to all this, the devices do not use the resources of your servers, as they have their own processors and memory, and the host for the phones should not be super-powerful, the “home” desktop will cope with all this. Compare the pros and cons, consider what is more profitable - surely in your automated testing system there is a place for real devices. I wish you fewer bugs and more test coverage. :)