Grouping Android phone models by Docker containers

A bit of background

The mobile application Badoo exists for the main “native” platforms (Android, iOS and Windows Phone) and for the mobile web. Despite the fact that in development we do not use any cross-platform frameworks, the vast majority of business logic in applications is similar, and in order not to duplicate functional tests for all platforms, we write cross-platform tests using Cucumber, Calabash and Appium. This allows us to make the code in the common part and reuse in tests for all platforms that are responsible for checking this very business logic. The only thing that remains is the implementation of interaction with the application (in more detail we told about it here ).

When cross-platform automation was just beginning (on iOS and Android), it was decided to use Mac Mini servers as servers. This made it possible to make each of the 8 build machines universal: it was possible to build and run functional and unit tests on it for both iOS and Android applications. This solution suited us practically to all until the number of functional tests exceeded five hundred for each platform, and the runs did not require more time. In order to keep the runtime within reasonable limits, we are constantly working on test optimization, as well as adding new Android devices (for iOS, we add simulators in a different way). Over time, we have a Mac Mini with more than 8 smartphones. It is important to note that we connect devices of the same model to the same server so that the test runs are consistent on the same agent.

Essentially

In Badoo, we decided to move testing of Android devices to Linux hosts - the necessary equipment is cheaper, and in addition, Mac Mini computers used for assembly often interrupt USB connections to Android devices, and they suddenly disappear during testing. To manage Linux servers, we mainly use Docker containers, so we decided to try to create a container for testing real Android devices and clone it for each model or group of phones in order to integrate the container into the existing server configuration.

A quick note: one of the advantages of Linux over Mac is that Linux is an open system. She showed us that the reason for the mysterious disappearance of phones when testing lies in the disconnection of connections lasting a split second. We fixed the tests by adding a new connection attempt to them, which largely solved the problem.
')

Essentially: Docker

Docker is a system that contains methods for building and distributing software configurations with an operating system infrastructure that isolates each software container from the rest of the computer. The container has its own file system, address space, etc. The containers are executed in one instance of the operating system, but since the system isolates processes more strongly, it all works as a set of virtual machines.

Explaining charts published on the Docker website:

The host computer uses a virtualization system running guest OS instances:


Docker containers run on the same OS:

Essentially: adb / adbd grouping

Each container had to manage its own set of phones. To implement this in the most natural way, you need to map the USB socket groups to different containers. Devices connected to the connectors on the front of the computer appear in the / dev / bus / usb / 001 directory, which is accessible to container 1; The devices connected to the connectors on the rear panel appear in the / dev / bus / usb / 002 directory, which is available to container 2. To increase the number of connected devices, an additional expansion card was ordered.
It all looks good, but the adb command communicates with the phone through a daemon, which uses the default port 5037 and works at the level of the entire computer. This means that the first container in which the adb command is executed starts the adb (adbd) daemon - as a result, the remaining containers connected to this daemon are seen by the phones of the first container. This problem could be solved using Docker's network capabilities (each container gets its own IP address, and, therefore, its own set of ports), but we used a different mechanism. For each container, a separate environment variable ANDROID_ADB_SERVER_PORT was assigned. We have allocated a port to each container so that it runs its own adb daemon, which sees only the phones of this container.

During the development process, we realized that we could not execute the adb command at the host level without setting the ANDROID_ADB_SERVER_PORT variable, since the host-level adbd daemon, capable of seeing all the USB ports, “steals” phones from Docker containers (phones can interact with only one adbd daemon each moment of time).
If we only used emulators, it would be possible to get by with separate adbd processes, but since we are working with real devices ...

Essentially: update containers with hot-plug USB devices

The second problem (and the main reason for writing this article) was that when the phone rebooted during the normal build procedure, it disappeared from the file system and the container phone list and did not appear again!

You can track the addition and removal of phones on the host computer by files in the / dev / bus / usb directory, in which the system creates and deletes files that correspond to the phones:

while sleep 3; do find /dev/bus/usb > /tmp/a diff /tmp/a /tmp/b mv /tmp/a /tmp/b done 

Unfortunately, phones in Docker containers are not only not being created or removed in this way; if you configure the creation and deletion of nodes, they do not actually interact with the phones!

We solved this question "in the forehead": we placed the containers in the --privileged mode and opened them access to the entire directory / dev / bus / usb.

Now I needed another mechanism for distributing phones to interface buses. I downloaded the Android source code and made small changes to the platform / system / core / adb / usb_linux.cpp file

  std::string bus_name = base + "/" + de->d_name; + const char* filter = getenv("ADB_DEV_BUS_USB"); + if (filter && *filter && strcmp(filter, bus_name.c_str())) continue; std::unique_ptr<DIR, int(*)(DIR*)> dev_dir(opendir(bus_name.c_str()), closedir); if (!dev_dir) continue; 

The variable ADB_DEV_BUS_USB is assigned a separate value for each container, indicating the bus with which the container should operate.

Retreat: although the fix was a simple one, the adb assembly had to be done through trial and error, since most people include everything in the assembly. My final decision looked like this (in a case-sensitive file system - I work on a Mac):

 cd src/android-src source build/envsetup.sh lunch 6 vi system/core/adb/usb_linux.cpp JAVA_NOT_REQUIRED=true make adb out/host/linux-x86/bin/adb 

Essentially: USB port multiplexing

Things were going well, but after installing the USB expansion board, we found that there was only one USB bus — there are now three buses on the computer, and we have five groups of devices.

Since I already got into the adb code, I decided to just add another environment variable: the ADB_VID_PID_FILTER variable gets the list of vid: pid identifier pairs, and adb ignores any nonconforming devices.

The fix is ​​below. When scanning the USB bus to detect connected phones, adbd processes may enter a race condition, but in practice this causes no problems.

 diff --git a/adb/usb_linux.cpp b/adb/usb_linux.cpp index 500898a..92e15ca 100644 --- a/adb/usb_linux.cpp +++ b/adb/usb_linux.cpp @@ -115,6 +115,71 @@ static inline bool contains_non_digit(const char* name) { return false; } +static int iterate_numbers(const char* list, int* rejects) { + const char* p = list; + char* end; + int count = 0; + while(true) { + long value = strtol(p, &end, 16); +//printf("%d, %p ... %p (%c) = %ld (...%s)\n", count, p, end, *end, value, p); + if (p == end) return count; + p = end + 1; + count++; + if (rejects) rejects[count] = value; + if (!*end || !*p) return count; + } +} + +int* compute_reject_filter() { + char* filter = getenv("ADB_VID_PID_FILTER"); + if (!filter || !*filter) { + filter = getenv("HOME"); + if (filter) { + const char* suffix = "/.android/vidpid.filter"; + filter = (char*) malloc(strlen(filter) + strlen(suffix) + 1); + *filter = 0; + strcat(filter, getenv("HOME")); + strcat(filter, suffix); + } + } + if (!filter || !*filter) { + return (int*) calloc(sizeof(int), 1); + } + if (*filter == '.' || *filter == '/') { + FILE *f = fopen(filter, "r"); + if (!f) { + if (getenv("ADB_VID_PID_FILTER")) { + // Only report failure for non-default value + printf("Unable to open file '%s'\n", filter); + } + return (int*) calloc(sizeof(int), 1); + } + fseek(f, 0, SEEK_END); + long fsize = ftell(f); + fseek(f, 0, SEEK_SET); //same as rewind(f); + filter = (char*) malloc(fsize + 1); // Yes, it's a leak. + fsize = fread(filter, 1, fsize, f); + fclose(f); + filter[fsize] = 0; + } + int count = iterate_numbers(filter, 0); + if (count % 2) printf("WARNING: ADB_VID_PID_FILTER contained %d items\n", count); + int* rejects = (int*)malloc((count + 1) * sizeof(int)); + *rejects = count; + iterate_numbers(filter, rejects); + return rejects; +} + +static int* rejects = 0; +static bool reject_this_device(int vid, int pid) { + if (!*rejects) return false; + for ( int len = *rejects; len > 0; len -= 2 ) { +//printf("%4x:%4x vs %4x:%4x\n", vid, pid, rejects[len - 1], rejects[len]); + if ( vid == rejects[len - 1] && pid == rejects[len] ) return false; + } + return true; +} + static void find_usb_device(const std::string& base, void (*register_device_callback) (const char*, const char*, unsigned char, unsigned char, int, int, unsigned)) @@ -127,6 +192,8 @@ static void find_usb_device(const std::string& base, if (contains_non_digit(de->d_name)) continue; std::string bus_name = base + "/" + de->d_name; + const char* filter = getenv("ADB_DEV_BUS_USB"); + if (filter && *filter && strcmp(filter, bus_name.c_str())) continue; std::unique_ptr<DIR, int(*)(DIR*)> dev_dir(opendir(bus_name.c_str()), closedir); if (!dev_dir) continue; @@ -176,6 +243,12 @@ static void find_usb_device(const std::string& base, pid = device->idProduct; DBGX("[ %s is V:%04x P:%04x ]\n", dev_name.c_str(), vid, pid); + if(reject_this_device(vid, pid)) { + D("usb_config_vid_pid_reject"); + unix_close(fd); + continue; + } + // should have config descriptor next config = (struct usb_config_descriptor *)bufptr; bufptr += USB_DT_CONFIG_SIZE; @@ -574,6 +647,7 @@ static void register_device(const char* dev_name, const char* dev_path, static void device_poll_thread(void*) { adb_thread_setname("device poll"); D("Created device thread"); + rejects = compute_reject_filter(); while (true) { // TODO: Use inotify. find_usb_device("/dev/bus/usb", register_device); 

I hope that my ideas will be useful to you if you implement a similar project. Leave questions in the comments below.

Tim Baverstock
QA automation engineer