BYOD in a container: we virtualize Android. Part one
On Friday, a video was published on Habré about how virtualization works on Android smartphones. It was developed and brought to the prototype stage at Parallels Labs by two students of the Department of MIT of the Academic University of St. Petersburg as part of their master's work. I was fortunate enough to find out what technology was under the hood, and also to ask the project participants what problems they were solving, how they overcame their difficulties and what they came up with as a result. The review is planned in two parts. In this post there will be a short overview of the existing solutions for virtualization on Android, clear diagrams of the architecture of our solution, a short video of how everything works. In the second part there will be more specifics. We will talk about the virtualization of the telephone part of smartphones, sound subsystem and input system.
I will show that our solution is under the hood. I will tell you what tasks the development team was solving, how the difficulties encountered were overcome and what result was achieved. The article consists of two parts. In the first (under the cut) there will be a short overview of existing solutions for virtualization on Android, clear schemes of the solution architecture, a short video of how everything works.
')
The concept of Bring Your Own Device (or BYOD), when an employee is allowed to work with the most convenient “iron” for him, is a double-edged sword. On the one hand, it is convenient for the user to use in the service what he is used to using at home. This theoretically increases his productivity. On the other hand, BYOD adds a headache to system administrators and IT directors who have to somehow solve the problem of protecting corporate applications and data that are now spinning on a smartphone. One of the main pitfalls is user behavior. Often, users ignore the ban on downloading and installing demo versions of third-party software, although a virus or a Trojan may be hidden inside such applications.
The obvious way to protect against this type of threat is to create a special sandbox for unreliable software; or, on the contrary, create such a “sandbox” for corporate software and business data. Both approaches should be implemented delicately, that is, without encroachment on private (SMS, pictures from parties, etc.) information of the employee. Such "sandboxes" are found very often in the software development and testing industry and are created using virtualization. We decided to do something similar, but - on smartphones.
Before diving into technical details, consider the entire zoo of existing solutions and virtualization technologies. This is necessary in order to understand what specific approach might be interesting for the implementation of our technology. The first known attempts to virtualize mobile devices began in 2008, and to date there are several successful approaches and projects. Of course, we considered the most mature and viable approaches. All of them are collected for clarity on the one in the picture, about each will be a few words separately.
Cells is the first project to create container virtualization technology for the Android platform. (We’ll take a closer look at what containers are and why Android is most conveniently virtualized using containers.) The technology provides simultaneous access of Android’s simultaneously running environments to input devices, a graphics accelerator, a cellular communication module and network devices, and also allows you to limit each container’s access system device. Measurements show that the requirements of this technology for computing resources, as well as the battery consumption of the device components of this technology are negligible.
VMware Horizon Mobile is a full-featured hypervisor of the second type, virtualizing the hardware of an abstract machine. Under the project, a Linux kernel with Android patches was created, on top of which the user-created Android environment created by VMware is launched. The hypervisor runs on common types of ARM processors that do not have extensions for hardware virtualization, and is a process in the host (main) OS - for example, in the native Android device OS. Obviously, in this case, the virtualization of all the hardware requires a lot of computing resources. But, more importantly, running more than one hypervisor will “land” the battery too quickly. No one needs a tablet discharged in half an hour. In addition, the approach of VMware seemed significantly more difficult to implement than the approach of Cells.
TrustDroid is a prototype of the application isolation system for the Android platform based on trust domains. Each application (including standard Android applications) is assigned a domain. Applications from different domains cannot interact with each other and cannot work with data published by applications from other domains. TrustDroid does not use hardware or user space virtualization. To support domain policy, changes are made to the kernel and to standard Android system applications. This system is the most undemanding to resources compared to the previous two.
The implementation of BYOD scripts and sandbox creation is different: in the case of Cells and VMware, policies are configured at the level of Android user environments; when using TrustDroid, the user needs to manually configure policies and domains for each application. This is more complicated and dreary for the user, because there may be several user environments. TrustDroid doesn't have everything OK with security. It is assumed that the standard Android system applications and the OS kernel are never compromised, but this may be wrong, because when you get superuser rights, you can replace them.
Disadvantages can be overcome by significantly improving TrustDroid. However, TrustDroid only solves the task of ensuring security on the device, while the user is also important the possibility of data isolation, located in different environments.
EmbeddedXEN is a project for porting XEN to the ARM platform. The project, as far as I know, has not yet come out of the stage of active development, and therefore cannot be considered as a complete solution. Currently, the project has already adapted the Linux version of the HTC Desire HD Linux kernel to work as a Dom0.
LXC is an add-on to the Linux kernel namespace infrastructure. It does not cover all the needs that arise when creating fully insulated containers. For example, LXC does not provide sysfs virtualization mechanisms and standard devices, such as a virtual terminal. The OpenVZ technology provides all the necessary features for creating fully insulated containers and is widely used by hosts. Therefore, the first thing the developers tried to port it to the Linux kernel used in Android.
Porting such a number of changes to the 2.6.35 kernel seemed excessive to us. But, abandoning OpenVZ, we lost the sysfs virtualization solution: when creating a new virtual environment, OpenVZ copies the tree of system objects (kobject) of the root user environment, which required additional efforts from us.
So, it was decided to use standard Linux-containers LXC and the entire infrastructure of the kernel, responsible for the implementation of namespaces (namespaces) and the delimitation of groups of resources.
The figure shows the architecture of our solution.
The binder kernel driver is an IPC mechanism developed by the AOSP project. The presence of the original, different from the standard IPC is not completely clear, but the details of the binder implementation were one of the obstacles to running several Android virtual environments. The problem was as follows. This driver has static variables, the values of which can be set by the ioctl system call. This system call is made by the servicemanager program, which is launched during the Android initialization process. The binder driver checks whether these variables are initialized and, if so, returns an error, which leads to the end of the servicemanager program, and since this program is necessary for Android to work, it also stops the initialization of Android.
Thus, it was necessary to create instances of the variables mentioned above for each container and access the virtual state instance belonging to the container in the context of which the ioctl () system call was made.
The big challenge with Android virtualization is peripheral virtualization and multiplexing of data streams. Almost everywhere we used the same approach, which can be called state virtualization.
Android assumes that it uses peripherals exclusively. The drivers for these devices are often written on the assumption that only one OS uses them. When running multiple Android on the same device in the peripheral devices, their drivers, the Android software stack, critical errors occur.
Thus, for the operation of several Android in containers it is required to ensure the possibility of simultaneous use of peripheral devices by all OSes.
The main thing that is required for this is to be able to control Android access to peripheral devices so that they cannot use peripheral devices arbitrarily.
Next, we will look at how the graphic screen will be virtualized, and in the next article we will talk about telephony, sound and touchscreen virtualization.
The need to separate the access of containers to the screen is obvious. When running multiple containers, each one starts displaying its own UI on the screen. Naturally, if you do not agree on which container has the exclusive right to use the screen, then on the screen you get a mixture of pixels and interface elements. To prevent this from happening, the screen must be virtualized. It is necessary that the physical device was replaced by a virtual one.
According to the Android Porting Guide , a driver like Linux Framebuffer is used to access the screen, and for porting you need to implement this driver. Linux framebuffer has a simple interface consisting of two dozen functions, most of which have a default implementation. The driver is used from user space.
In addition to the framebuffer driver, the GPU has access to the physical screen. In order for inactive OSes not to display their UI on the screen, it is required to change the operation of displaying their frames. It turned out that in the case of the experimental Google Nexus S and Samsung Galaxy SII, the smartphone screens copy the frame for display from RAM using DMA. The framebuffer driver informs the screen of the address where the frame is located to be displayed (screen memory). Thus, the operation of displaying a frame on the screen looks like the operation of recording a frame in the device's RAM at a specific address.
To replace a write operation in RAM, it is necessary to establish by whom it can be performed. In the case of graphics processing, the GPU or CPU can perform frame recording. Modern GPU and CPU are composed of MMU. To ensure that inactive Android frames are not recorded in the screen's memory, it is enough to change the display of addresses in the MMU so that the addresses of inactive Android displayed in the screen's memory actually point to the usual buffer in the device's RAM. Let's call it the shadow buffer.
Android renders its UI as it changes, not by timer. Therefore, if you simply redirect the MMU to the screen memory when switching active Android, then active Android may not redraw the frame on the screen and the image on it will be from the previous active OS.
If you select a shadow buffer for each Android, then when you switch the active OS, just copy the frame on the screen to the shadow buffer of the previous active Android. To display Android, which has become active, you need to copy the contents of its shadow buffer into the screen's memory. But the shadow buffer takes from 2 to 4 MB of physical memory. Having the ability to cause a redraw of the frame when switching active Android, you can get rid of the selection of the shadow buffer for each Android, leaving a single shadow buffer.
This feature is provided by the fb_early_suspend mechanism. The shadow buffer now becomes not a place for storing the Android frame, but a place where all inactive OSs write their picture. In addition, due to the presence of the MMU, you can reduce the size of the shadow buffer to one page of physical memory by mapping all 2–4 MB of addresses into a single page.
A typical scenario of using the Linux Framebuffer driver is to map the screen's memory to the user process address space with the mmap () system call and standard and special IOCTL calls.
Standard IOCTL are handled by the standard Linux Framebuffer IOCTL handler in the OS kernel. Special IOCTLs have non-standard semantics and can only be processed by the physical Linux Framebuffer driver. Since the physical Linux Framebuffer was replaced with a virtual one, mmap () and IOCTL are called from the virtual Linux Framebuffer driver.
All mmap () calls are processed completely in the virtual Linux Framebuffer driver without using the physical Linux Framebuffer driver, since The semantics of this system call are standard. Active Android processes call mmap () to access screen memory. Processes of inactive Android when calling mmap () access the shadow buffer instead of screen memory. For all running Androids, a virtual state of the Linux Framebuffer driver is created. For active Android, all calls to the virtual Linux Framebuffer driver are immediately redirected to the physical Linux Framebuffer driver, which changes the state of the physical Linux Framebuffer driver. For inactive Android, standard IOCTLs change their virtual states of the Linux Framebuffer. Non-standard IOCTLs for inactive Android return an error. On the Google Nexus S and Samsung Galaxy SII, that was enough.
Android virtualization technology built on Linux containers has proven to work on several models of smartphones. We were able to achieve the simultaneous execution of any applications simultaneously with sound mixing and receiving incoming messages and calls, as each of their virtualized phones would do. For example, our demo script shows that we are simultaneously playing Andgy Birds and playing music in different containers. This is how it happens if someone has not seen.
How sound, telephony and user input virtualization was implemented in the next article .
I will show that our solution is under the hood. I will tell you what tasks the development team was solving, how the difficulties encountered were overcome and what result was achieved. The article consists of two parts. In the first (under the cut) there will be a short overview of existing solutions for virtualization on Android, clear schemes of the solution architecture, a short video of how everything works.
')
Introduction: Why is all this necessary?
The concept of Bring Your Own Device (or BYOD), when an employee is allowed to work with the most convenient “iron” for him, is a double-edged sword. On the one hand, it is convenient for the user to use in the service what he is used to using at home. This theoretically increases his productivity. On the other hand, BYOD adds a headache to system administrators and IT directors who have to somehow solve the problem of protecting corporate applications and data that are now spinning on a smartphone. One of the main pitfalls is user behavior. Often, users ignore the ban on downloading and installing demo versions of third-party software, although a virus or a Trojan may be hidden inside such applications.
The obvious way to protect against this type of threat is to create a special sandbox for unreliable software; or, on the contrary, create such a “sandbox” for corporate software and business data. Both approaches should be implemented delicately, that is, without encroachment on private (SMS, pictures from parties, etc.) information of the employee. Such "sandboxes" are found very often in the software development and testing industry and are created using virtualization. We decided to do something similar, but - on smartphones.
We are looking for analogues
Before diving into technical details, consider the entire zoo of existing solutions and virtualization technologies. This is necessary in order to understand what specific approach might be interesting for the implementation of our technology. The first known attempts to virtualize mobile devices began in 2008, and to date there are several successful approaches and projects. Of course, we considered the most mature and viable approaches. All of them are collected for clarity on the one in the picture, about each will be a few words separately.
Cells is the first project to create container virtualization technology for the Android platform. (We’ll take a closer look at what containers are and why Android is most conveniently virtualized using containers.) The technology provides simultaneous access of Android’s simultaneously running environments to input devices, a graphics accelerator, a cellular communication module and network devices, and also allows you to limit each container’s access system device. Measurements show that the requirements of this technology for computing resources, as well as the battery consumption of the device components of this technology are negligible.
VMware Horizon Mobile is a full-featured hypervisor of the second type, virtualizing the hardware of an abstract machine. Under the project, a Linux kernel with Android patches was created, on top of which the user-created Android environment created by VMware is launched. The hypervisor runs on common types of ARM processors that do not have extensions for hardware virtualization, and is a process in the host (main) OS - for example, in the native Android device OS. Obviously, in this case, the virtualization of all the hardware requires a lot of computing resources. But, more importantly, running more than one hypervisor will “land” the battery too quickly. No one needs a tablet discharged in half an hour. In addition, the approach of VMware seemed significantly more difficult to implement than the approach of Cells.
TrustDroid is a prototype of the application isolation system for the Android platform based on trust domains. Each application (including standard Android applications) is assigned a domain. Applications from different domains cannot interact with each other and cannot work with data published by applications from other domains. TrustDroid does not use hardware or user space virtualization. To support domain policy, changes are made to the kernel and to standard Android system applications. This system is the most undemanding to resources compared to the previous two.
The implementation of BYOD scripts and sandbox creation is different: in the case of Cells and VMware, policies are configured at the level of Android user environments; when using TrustDroid, the user needs to manually configure policies and domains for each application. This is more complicated and dreary for the user, because there may be several user environments. TrustDroid doesn't have everything OK with security. It is assumed that the standard Android system applications and the OS kernel are never compromised, but this may be wrong, because when you get superuser rights, you can replace them.
Disadvantages can be overcome by significantly improving TrustDroid. However, TrustDroid only solves the task of ensuring security on the device, while the user is also important the possibility of data isolation, located in different environments.
EmbeddedXEN is a project for porting XEN to the ARM platform. The project, as far as I know, has not yet come out of the stage of active development, and therefore cannot be considered as a complete solution. Currently, the project has already adapted the Linux version of the HTC Desire HD Linux kernel to work as a Dom0.
Container virtualization is the answer.
LXC is an add-on to the Linux kernel namespace infrastructure. It does not cover all the needs that arise when creating fully insulated containers. For example, LXC does not provide sysfs virtualization mechanisms and standard devices, such as a virtual terminal. The OpenVZ technology provides all the necessary features for creating fully insulated containers and is widely used by hosts. Therefore, the first thing the developers tried to port it to the Linux kernel used in Android.
- The current experimental version of the OpenVZ patch is for kernel version 2.6.32, but on smartphones on which the technology was developed and tested, kernels version 2.6.35 work;
- when trying to apply an OpenVZ patch to one of the used cores, it was not possible to patch 166 files from 1682; in addition, the patch modifies 34 files specific to the x86 architecture.
Porting such a number of changes to the 2.6.35 kernel seemed excessive to us. But, abandoning OpenVZ, we lost the sysfs virtualization solution: when creating a new virtual environment, OpenVZ copies the tree of system objects (kobject) of the root user environment, which required additional efforts from us.
So, it was decided to use standard Linux-containers LXC and the entire infrastructure of the kernel, responsible for the implementation of namespaces (namespaces) and the delimitation of groups of resources.
The figure shows the architecture of our solution.
- Control Panel. Graphic interface to LXC; allows you to manage containers (start, stop, switch the active container).
- Supervisor kernel, in the figure - AndCont supervisor. Initializes the virtual state of the drivers, intercepting the moment the container is started, implements switching between containers.
- The mechanism of intercontainer interaction (ICC). A means of communication between containers based on Netlink, bypassing checks in the applied network protocols and based on shared memory.
- Input manager Needed to share Android input devices running in parallel.
- Virtual framebuffer driver. Provides the ability to simultaneously use the screen of the device.
- GPU driver. Implements the possibility of simultaneous use of the GPU by all containers.
- Binder. IPC driver, made in the framework of the Android Open Source Project (AOSP).
- Alarm Interface to the clock.
- Proxy server for Radio Interface Management (Radio Interface Layer, RIL). Provides sharing of multiple Android containers to mobile network access equipment.
- Proxy client RIL. Accepts calls to the telephony service from the applications in the container and sends them to the RIL proxy server.
- Proxy client audio. Gains control of the container's audio streams and transfers them to the audio proxy server.
- Proxy audio server. Reproduces the audio streams of containers.
- Auxiliary container. Designed to run system daemons and Android libraries, for various reasons rendered from the Android guest.
Container Management Mechanism
- The reset of the CAP_SYS_BOOT flag, which is responsible for the ability to reboot the system, which is used by the Dalvik java virtual machine during its initialization, is blocked.
- Checks of open file descriptors other than the standard ones in lxc_start are omitted.
- Fixed a bug that caused file descriptors to leak from the adbd daemon, which is used to access the smartphone via USB.
- Added kernel-level supervisor notifications about launching a new container before launching the init process.
Binder
The binder kernel driver is an IPC mechanism developed by the AOSP project. The presence of the original, different from the standard IPC is not completely clear, but the details of the binder implementation were one of the obstacles to running several Android virtual environments. The problem was as follows. This driver has static variables, the values of which can be set by the ioctl system call. This system call is made by the servicemanager program, which is launched during the Android initialization process. The binder driver checks whether these variables are initialized and, if so, returns an error, which leads to the end of the servicemanager program, and since this program is necessary for Android to work, it also stops the initialization of Android.
Thus, it was necessary to create instances of the variables mentioned above for each container and access the virtual state instance belonging to the container in the context of which the ioctl () system call was made.
Peripheral Virtualization
The big challenge with Android virtualization is peripheral virtualization and multiplexing of data streams. Almost everywhere we used the same approach, which can be called state virtualization.
Android assumes that it uses peripherals exclusively. The drivers for these devices are often written on the assumption that only one OS uses them. When running multiple Android on the same device in the peripheral devices, their drivers, the Android software stack, critical errors occur.
Thus, for the operation of several Android in containers it is required to ensure the possibility of simultaneous use of peripheral devices by all OSes.
The main thing that is required for this is to be able to control Android access to peripheral devices so that they cannot use peripheral devices arbitrarily.
- The physical device is replaced with a virtual one, so that all Android requests to the physical device are sent to the virtual device.
- The virtual device performs part of the Android requests on the physical device, and the part is virtual, for example, changing the virtual state in accordance with its policy.
- The virtual device has the same interface and behavior as the physical one.
- The interface should be as compatible with all available versions of Android;
- The interface must be unique and used for all tasks accessing the device.
Next, we will look at how the graphic screen will be virtualized, and in the next article we will talk about telephony, sound and touchscreen virtualization.
Display
The need to separate the access of containers to the screen is obvious. When running multiple containers, each one starts displaying its own UI on the screen. Naturally, if you do not agree on which container has the exclusive right to use the screen, then on the screen you get a mixture of pixels and interface elements. To prevent this from happening, the screen must be virtualized. It is necessary that the physical device was replaced by a virtual one.
According to the Android Porting Guide , a driver like Linux Framebuffer is used to access the screen, and for porting you need to implement this driver. Linux framebuffer has a simple interface consisting of two dozen functions, most of which have a default implementation. The driver is used from user space.
Reprogramming MMU GPU
In addition to the framebuffer driver, the GPU has access to the physical screen. In order for inactive OSes not to display their UI on the screen, it is required to change the operation of displaying their frames. It turned out that in the case of the experimental Google Nexus S and Samsung Galaxy SII, the smartphone screens copy the frame for display from RAM using DMA. The framebuffer driver informs the screen of the address where the frame is located to be displayed (screen memory). Thus, the operation of displaying a frame on the screen looks like the operation of recording a frame in the device's RAM at a specific address.
To replace a write operation in RAM, it is necessary to establish by whom it can be performed. In the case of graphics processing, the GPU or CPU can perform frame recording. Modern GPU and CPU are composed of MMU. To ensure that inactive Android frames are not recorded in the screen's memory, it is enough to change the display of addresses in the MMU so that the addresses of inactive Android displayed in the screen's memory actually point to the usual buffer in the device's RAM. Let's call it the shadow buffer.
Shadow Framebuffer
Android renders its UI as it changes, not by timer. Therefore, if you simply redirect the MMU to the screen memory when switching active Android, then active Android may not redraw the frame on the screen and the image on it will be from the previous active OS.
If you select a shadow buffer for each Android, then when you switch the active OS, just copy the frame on the screen to the shadow buffer of the previous active Android. To display Android, which has become active, you need to copy the contents of its shadow buffer into the screen's memory. But the shadow buffer takes from 2 to 4 MB of physical memory. Having the ability to cause a redraw of the frame when switching active Android, you can get rid of the selection of the shadow buffer for each Android, leaving a single shadow buffer.
This feature is provided by the fb_early_suspend mechanism. The shadow buffer now becomes not a place for storing the Android frame, but a place where all inactive OSs write their picture. In addition, due to the presence of the MMU, you can reduce the size of the shadow buffer to one page of physical memory by mapping all 2–4 MB of addresses into a single page.
Linux Virtual Framebuffer Driver
A typical scenario of using the Linux Framebuffer driver is to map the screen's memory to the user process address space with the mmap () system call and standard and special IOCTL calls.
Standard IOCTL are handled by the standard Linux Framebuffer IOCTL handler in the OS kernel. Special IOCTLs have non-standard semantics and can only be processed by the physical Linux Framebuffer driver. Since the physical Linux Framebuffer was replaced with a virtual one, mmap () and IOCTL are called from the virtual Linux Framebuffer driver.
All mmap () calls are processed completely in the virtual Linux Framebuffer driver without using the physical Linux Framebuffer driver, since The semantics of this system call are standard. Active Android processes call mmap () to access screen memory. Processes of inactive Android when calling mmap () access the shadow buffer instead of screen memory. For all running Androids, a virtual state of the Linux Framebuffer driver is created. For active Android, all calls to the virtual Linux Framebuffer driver are immediately redirected to the physical Linux Framebuffer driver, which changes the state of the physical Linux Framebuffer driver. For inactive Android, standard IOCTLs change their virtual states of the Linux Framebuffer. Non-standard IOCTLs for inactive Android return an error. On the Google Nexus S and Samsung Galaxy SII, that was enough.
Intermediate conclusions
Android virtualization technology built on Linux containers has proven to work on several models of smartphones. We were able to achieve the simultaneous execution of any applications simultaneously with sound mixing and receiving incoming messages and calls, as each of their virtualized phones would do. For example, our demo script shows that we are simultaneously playing Andgy Birds and playing music in different containers. This is how it happens if someone has not seen.
How sound, telephony and user input virtualization was implemented in the next article .
Source: https://habr.com/ru/post/174211/
All Articles