How Android works, part 4

Hello! We took the time to continue the series of articles about the internal device Android. In this article, I will discuss the Android boot process, the file system contents, how user and application data is stored, root access, the portability of Android assemblies, and the problem of fragmentation.

Articles series:

• How Android works, part 1
• How Android works, part 2
• How Android works, part 3
• How Android works, part 4
• ...

Packages

As I said before, the Android architecture is built around applications. It is applications that play a key role in the device of many parts of the system, the model of activity and intent is built for the harmonious interaction of applications, and the Android security model is based on the isolation of applications. And if the activity manager is in charge of orchestrating the interaction of application components, then the package manager is responsible for installing, updating, and managing application rights (package manager — in the shell, you can call it with the pm command).

As the name “package manager” suggests, at this level applications are often called packages . Packages are distributed in the format of APK (Android package) - special zip-archives. Each package has a name (also known as an application ID ) that uniquely identifies this application (but not its specific version — on the contrary, the names of different versions of the package must match, otherwise they will be considered separate packages). Packet names are usually written in reverse DNS name notation - for example, the YouTube application uses the package name com.google.android.youtube . Often, the package name is the same as the namespace used in its Java code, but Android does not require it (besides, the application APK files usually include third-party libraries, whose namespace naturally has nothing to do with the package names, who use them).

Each APK must be signed by the developer using a digital signature when building. Android checks for the presence of this signature when installing the application, and when updating an already installed application, it additionally compares the public keys with which the old and new versions are signed; they must match, which ensures that the new version was created by the same developer as the old one. (If this check were not available, the attacker could create a package with the same name as the existing application, convince the user to install it by “updating” the application, and access the data of this application.)

The package update itself is the installation of its new version instead of the old one, preserving the data and permissions received from the user. It is also possible to “roll back” (downgrade) applications to older versions, but by default Android deletes the data saved by the new version, since the old version may not be able to work with the data formats used by the new version.

As I said, usually the code of each application is executed under its own Unix-user (UID), which ensures their mutual isolation. Multiple applications can explicitly ask Android to use a common UID for them, which will allow them to directly access each other’s files and even, if they wish, run in the same process.

Although usually one package corresponds to one APK file, Android supports packages consisting of several APKs (this is called split APK, or split APK ). This is the basis of such “magic” features of Android, such as dynamic loading of additional modules of the application ( dynamic feature modules ) and Instant Run in Android Studio (automatically updating the code of the running application without reinstalling it completely and, in many cases, even without restarting).

File system

The device of the file system is one of the most important and interesting questions in the architecture of the operating system, and the device of the file system in Android is no exception.

Of interest is, firstly, which file systems are used, that is, in what format the contents of the files are stored on the conditional disk (in the case of Android, this is usually flash-memory and SD-cards) and how this format is supported by . The Linux kernel used in Android to a varying degree supports a large number of very different file systems - from the FAT and NTFS used in Windows and the infamous HFS + and modern APFS used in Darwin - to the network 9pfs from Plan 9. There are many "native "For Linux file systems - for example, Btrfs and the ext family.

The de facto standard for Linux for a long time has been ext4 , used by default by most popular Linux distributions. Therefore, there is nothing surprising in the fact that it is used in Android. Some builds (and some enthusiasts) also use F2FS (Flash-Friendly File System), optimized specifically for flash-memory (however, with its advantages, things are not so straightforward ).

Secondly, of interest is the so-called filesystem layout - the location of system and user folders and files in the file system. Filesystem layout in “normal Linux” deserves a more detailed description (which can be found, for example, by this link ); I will mention here only a few of the most important directories:

• /home stores users home folders; here, in various hidden folders ( .cache , .cache , .config and others), the programs store their settings, data and cache, specific to the user,
• /boot stores the Linux kernel and the initramfs image (special boot filesystem),
• /usr (it would be more logical to call /system ) stores the main part of the system itself, including libraries, executable files, configuration files, and also resources - themes for the interface, icons, the contents of the system manual, etc.,
• /etc (it would be more logical to call /config ) stores system-wide settings,
• /dev stores device files and other special files (for example, socket /dev/log ),
• /var stores changeable data - logs, system cache, database contents, etc.

Android uses a similar but noticeably different filesystem layout. Here are some of the most important parts of it:

• /data stores changeable data,
• the kernel and the initramfs image are stored on a separate flash partition (partition) that is not mounted on the main file system,
• /system matches /usr and stores the system,
• /vendor - analog /system designed for files specific to this Android build, and not included in the "standard" Android,
• /dev , as in “normal Linux”, stores device files and other special files.

The most interesting of these directories are /data and /system . The content /system describes the system and contains the majority of its component files. /system is located on a separate section of flash-memory, which by default is mounted in read-only mode; usually the data on it is changed only when the system is updated. /data also located on a separate section and describes the changing state of a specific device, including user settings, installed applications and their data, caches, etc. Clearing all user data, the so-called factory reset, with such a scheme is simply to clear the contents of the data section; The untouched system remains installed in the system section.

 # mount | grep /system /dev/block/mmcblk0p14 on /system type ext4 (ro,seclabel,relatime,data=ordered) # mount | grep /data /dev/block/mmcblk0p24 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,data=writeback) 

Applications — namely, their APK, odex files (compiled ahead-of-time Java code), and ELF libraries — are installed in /system/app (for applications delivered with the system) or in /data/app (for user-installed applications). When you create an Android assembly, each preinstalled application is allocated a folder with the name of the form /system/app/Terminal , and for user-installed applications, during installation, folders are created whose names begin with their package name. For example, the YouTube app is saved to a folder with a name like /data/app/com.google.android.youtube-bkJAtUtbTuzvAioW-LEStg==/ .

 # pm path com.android.egg package:/system/app/EasterEgg/EasterEgg.apk 

Since the pre-installed applications are stored in the system section (the contents of which, I recall, only change when the system is updated), they cannot be deleted (instead of this, Android provides the ability to "disable" them). Nevertheless, the update of the pre-installed applications is supported - at the same time a folder in /data/app is created for the new version, and the version supplied with the system remains in /system/app . In this case, the user has the opportunity to "remove updates" of such an application by returning to the version from /system/app .

Another feature of pre-installed applications is that they can receive special “system” permissions . For example, third-party applications do not get permissions DELETE_PACKAGES , which allows you to delete other applications, REBOOT , which allows you to restart the system, and READ_FRAME_BUFFER , which allows you to directly access the contents of the screen. These permissions have a signature protection level , that is, the application attempting to access them must be signed with the same key as the application or service in which they are implemented - in this case, since these permissions are implemented by the system, the key that signed itself build Android.

Each application is assigned a folder in /data/data (for example, /data/data/com.google.android.youtube for YouTube) for storing changeable data. Only the application itself has access to this folder — that is, only the UID under which this application runs (if the application uses multiple UIDs, or several applications use a common UID, things may be more complicated). In this folder, applications save the settings, the cache (in the subfolders of shared_prefs and cache respectively) and any other data they need:

 # ls /data/data/com.google.android.youtube/ cache code_cache databases files lib no_backup shared_prefs # cat /data/data/com.google.android.youtube/shared_prefs/youtube.xml <?xml version='1.0' encoding='utf-8' standalone='yes' ?> <map> <string name="user_account">username@gmail.com</string> <boolean name="h264_main_profile_supported7.1.2" value="true" /> <int name="last_manual_video_quality_selection_max" value="-2" /> <...> </map> 

The system is aware of the existence of the cache folder and can clean it itself if there is not enough space. When you delete an application, the entire folder of this application is completely deleted, and the application leaves no trace behind it. Alternatively, both can be explicitly done in the settings:

This data storage allocated to each application is called internal storage.

In addition, there is another type of storage in Android - the so-called external storage (external storage - this name reflects the original idea that the external storage should be located on the external SD card inserted into the phone). In fact, external storage plays the role of the user's home folder - it is there that folders such as Documents, Download, Music and Pictures are located, the external storage is opened by the file managers as the default folder, and it is the contents of the external Android storage that allows the computer to access when connected by cable.

 # ls /sdcard Alarms Download Podcasts Android Movies Ringtones Books Music Subtitles DCIM Notifications bluetooth Documents Pictures 

Unlike internal storage, divided into folders of individual applications, external storage is a “common zone”: it has full access to any application that has received permission from the user. As I mentioned in the last article, this permission should be requested by applications such as the file manager; and for most other applications, it is better to use intent with the ACTION_GET_CONTENT action, allowing the user to select the desired file in the system file manager.

Many applications prefer to save some of their internal files that are large (for example, the cache of downloaded images and audio files) in external storage. For this, Android allocates to the applications in the external storage a folder with the names of the type Android/data/com.google.android.youtube . The application itself does not need permission to access the entire external storage to access such a folder (because its UID is set as the owner of this folder), but any other application that has this permission can access this folder, so you really should use it only for storing public and non-critical data. When you delete an application, the system will delete its special folder in the external storage; but files created by applications in external storage outside their special folder are considered owned by the user and remain in place after the application that created them is deleted.

As I mentioned above, the initial assumption was that the external storage would actually be located on an external SD card, because at that time the volume of SD cards was significantly higher than the amount of memory embedded in the phones (in the very same HTC Dream there were only 256 megabytes, of which about 90 megabytes were allocated to the data section). Since then, many conditions have changed; in modern phones, there is often no slot for an SD card, but a huge amount of internal memory is installed by mobile standards (for example, in Samsung Galaxy Note 9 it can be up to 512 gigabytes ).

Therefore, in modern Android, almost always both internal and external storage are located in the internal memory. The actual path that the external storage is located in the file system is of the form /data/media/0 (for each user of the device , a separate external storage is created , and the number in the path corresponds to the user number). For compatibility purposes, the external storage can also be reached via the /sdcard , /mnt/sdcard , /storage/self/primary , /storage/emulated/0 paths, several paths starting with /mnt/runtime/ , and some others.

On the other hand, many devices still have a slot for an SD card. An SD card inserted into an Android device can be used as a regular external drive (without turning it into internal or external storage of the system) - save files to it, open files stored on it, use it to transfer files to other devices, etc. In addition, Android allows you to "borrow" the SD card and place the internal and external storage on it (this is called the adopted storage ). At the same time, the system reformats the SD card and encrypts its contents - the data stored on it cannot be read by connecting it to another device.

You can read more about all this, for example, in this post and in the official documentation for application developers and creators of Android assemblies .

Loading

The traditional approach to computer system security is limited to protecting the system from software attacks. It is believed that if the attacker has physical access to the computer, the game is already lost : he can get full access to any data stored on it. To do this, it is sufficient, for example, to launch an arbitrary operating system controlled by it on this computer, allowing it to bypass any restrictions imposed by the “primary” system, or directly connect the data disk to another device. If desired, an attacker can leave the computer in a healthy state, but patch the system installed on it, install arbitrary backdoors, keyloggers, etc.

The Unix user rights restriction model (and the app sandbox technology in Android based on it) is focused on protection against software attacks; the restriction of rights in Unix itself does not in any way protect the system from a user accessing the server and having physical access to the computer. And if serious multi-user servers can and should be protected from unauthorized physical access to personal computers - and even more so mobile devices - this approach is simply not applicable.

You can try to improve the situation with protection from an attacker who received physical access to the device in two ways:

• First, you can encrypt the data stored on the disk , thereby preventing the attacker from gaining access to the data itself, even if it has access to the contents of the disk.
• Secondly, it is possible in one way or another to limit the ability to load arbitrary operating systems on the device , forcing the attacker to undergo authentication and authorization procedures in the installed system.

It is with these two areas of protection that the secure boot model in Android is associated.

Verified Boot

The Android download process is designed so that, on the one hand, it does not allow attackers to download an arbitrary OS on the device, on the other hand, it can allow users to install customized Android assemblies (and other systems).

First of all, Android devices, unlike desktop computers, usually do not allow the user (or an attacker) to boot from external media; instead, the bootloader installed on the device starts up immediately. Bootloader is a relatively simple program whose tasks (when booting in ordinary mode) include:

• Initialization and configuration of the Trusted Execution Environment (for example, ARM TrustZone),
• Finding the internal memory partitions in which Linux kernel and initramfs images are stored,
• (integrity) — — ,
• initramfs .

Flashing, unlocking, fastboot recovery

, bootloader .

-, (Android) , recovery . recovery, Android- , , Android recovery .

«» bootloader', — ( flashing ) . bootloader , fastboot mode ( bootloader mode), bootloader' fastboot ( fastboot Android SDK ).

bootloader' . , Samsung, bootloader'a (Loke) (Odin). Odin Samsung ( Odin), Heimdall .

bootloader' ( ), recovery Android, , :

 $ fastboot flash recovery recovery.img $ fastboot flash boot boot.img $ fastboot flash system system.img 

; : , , bootloader' Android, , , , .

, bootloader' (unlocking the bootloader, OEM unlock) — bootloader' recovery, ( ). Android, LineageOS ( CyanogenMod ), Paranoid Android , AOKP , OmniROM .

bootloader' - , ( data ) . , , (, Google ), — , .

recovery bootloader , .

bootloader' :

recovery, «» (flashaholics) — TWRP (Team Win Recovery Project). - «», zip-, :

Android ( file-based encryption ). ext4 , Linux ( fscrypt ), .

, data , , (credential encrypted storage). , , . , , .

credential encrypted storage Android device encrypted storage — , ( Trusted Execution Environment). , , , . , Direct Boot : ; ( ) device encrypted storage, , . , Direct Boot , , - .

Root

root- — « root» (UID 0, ). , root — Unix-, — — , .

, Android , root- . , « », Android « », root- «», :

• — , APK ,
• : -, email-, , , , , , ( activity intent', ),
• (icon packs),
• , ,
• ( Android-) ,
• , Android, .

, , , root- . root- ( ), Android root . , , Unix-; shell, adb shell , Unix- shell .

, , root :

• -, root- Android — Android Studio QEMU, .
• -, root- Android ( pre-rooted ROMs).
• -, bootloader' root- Android , , su , bootloader.
• , -, , , root-, ( root- ). , 2016 , root- Dirty Cow .

? , root- . , root-, , , . — , . , , system ( ).

, root- — , .

root-

With great power comes great responsibility.

, . , root- .

, , . , root- , , , root- . root, , , , — , , , ..

, Unix, Android , — . root- , , . , root- Android, .

, , root- , . , , , , , . , — , Google Pay ( Android Pay) — root- , .

, - Pokémon GO root- , , .

Google Play root-

Google Google Play Store Google Play Services , root- , Android , , , , . Play Store — ( , ) Android, . ( — , Amazon Android Fire OS — Echo, Fire TV, Fire Phone, Kindle Fire Tablet — Google). root- Android-.

, Google , root-, Google Play ( Open GApps ); Google , .

Device manufacturers work with Google to certify that Android devices with Google apps installed are secure and will run apps correctly. To be certified, a device must pass Android compatibility tests. If you are unable to add a Google Account on your Android device, your Android device software might not have passed Android compatibility tests, or the device manufacturer has not submitted the results to Google to seek approval. As a result, your device is uncertified. This means that your device might not be secure.

If you are a User wanting to use custom ROMs on your device, please register your device by submitting your Google Services Framework Android ID below.

root

«» Linux, Unix-, root- ( sudo su ) ( , Unix- root, ) (, ). , root-.

su Android Open Source Project , Unix- shell ( root), root-. su , , , .

, su . su , root-. , SuperSU ; Magisk .

Magisk

Magisk — , , /system , system ( systemless-ly ), «» Linux . Magisk Hide — root-, root- Magisk — - Google Pay, root- . Magisk Hide root-, SafetyNet Google.

, , Google Pay root- — root- . , root-, - Google Pay. , .

Magisk systemless- «» Magisk Modules , Magisk. , , root- Xposed ViPER , .

SoC,

, «», , — , — (, Wi-Fi ).

, Android- (system on a chip, SoC). — , , -, , LTE-, Bluetooth- Wi-Fi- .. — . , , , , , .

, , . , «», , , , . , , -, , , , .

— Android — SoC . , , — LTE- — . , ; , , .

SoC (, Qualcomm) Android- (, Sony LG), Android, Android Open Source Project. , Android, , , .

Android Android-. , Android. Android , , . .

, : Android — , . Android, .

, , , Android. .

-, . , . , , , , .

-, Android — developer experience (DX, user experience/UX). , Android, API — Android Framework, OpenGL/Vulkan — . , , , Android — , , .

Don't Stop Thinking About Tomorrow

Android- . , Google Nexus Pixel , Android. Android , , , .

, , Android, SoC. Android «» (, , Android-x86 RemixOS ). Android ChromeOS, Chromebook' Android- Linux- -. — Android — Anbox , Android- «» Linux-. (, Android- x86-, , Java, .)

, Android — , Android .

— Android. , , . , , Android .

. 2017 Google Project Treble — , ( HAL, hardware abstraction layer) ( , ) . Treble , , , — — .

Treble . Treble (Sony, Nokia, OnePlus, Oppo, Xiaomi, Essential Vivo — Google) - Android Pie. Treble Essential Android Pie Essential Phone Android Pie. Android — - — , Treble, , SoC.

Treble . Java «write once, run everywhere» — Android- . Treble — , Android SoC. , , Treble. , Android- .

userspace Android: init, Zygote, Binder, props.