Integration of FATFS library for organizing reading of a disk device on iOS

Introduction

The article is devoted to the introduction of an open source library on iOS for reading / writing data from a MFI disk device based on FAT12 / FAT16 / FAT32 / Exfat. A method for building an application architecture based on the FATFS library, as well as methods for debugging and testing wired MFI devices, is presented. The article contains almost no code due to compliance with the NDA .

Formulation of the problem

The main goal was to create some kind of universal SDK (.framework), capable of working with manufacturers of various disk access devices using the MFI protocol, using a single API standard.



The figure shows a generalized block diagram of such a framework. Depending on the protocol of the device, this or that library of interaction with the device is selected. Depending on the file system (FS) used on the disk, the appropriate disk operation algorithm is selected: FAT / exFat / Other FS. At the same time, several applications (using this SDK) that are in the device’s memory can have access to the device; however, only one can perform writing / reading at a time. This SDK implies using it in a multi-threaded application with competitive read and write tasks.
')

Finding a solution

Initially, the implementation of FAT32, created by Apple, was chosen as the base library for the implementation of FS FAT32. However, the complexity of integrating the standard API for the access device due to being tied to a specific platform made it necessary to go in the direction of finding a ready solution with a separate abstract API for the physical access layer with the MFI device through the manufacturer's library. EFSL and FATFS open source implementations of FAT32 were also considered. FATFS was chosen for several reasons:

• current library support
• platform independence and ease of porting
• wide range of configuration options
• exFAT support
• abstract media driver access level

Due to the problems of obtaining logs of the application, given that the connector was physically busy and there was no possibility to connect to Xcode , the NSLogger library was connected, which allows you to transfer logs via Wi-Fi almost in real time mode. However, the problem of debugging such a library was the place to be. It was decided to develop a simulator MFI device, which accessed a certain area on an iOS / MAC device disk and emulated read / write using the same API as the real device.

Integration issues

Porting to Obj-C

FATFS is written in pure C, and it was necessary to integrate it into the library written in ObjC. Although ObjC directly supports the integration of C functions, there were problems with function signatures and using some types of variables.

Unicode support

The Unicode option worked only with the Obj-C NSUTF16LittleEndianStringEncoding encoding parameter, despite the fact that, according to the stated requirements, UTF-16BE and UTF-8 were additionally supported. All methods working with file names needed to be thread-safe for FAT32 to work correctly.

Thread safe

The thread safety option activated by the FS_REENTRANT parameter and implemented through the ff_req_grant, ff_rel_grant, ff_del_syncobj functions required additional implementation via POSIX. Without the thread safety option enabled, there was a damage to the file system table and, as a result, data loss. The pthread_t file descriptor was not immediately successfully implemented due to the fact that the ff_ functions for synchronizing threads passed data by value, not by reference. After replacing with reference values, there were no problems with the thread safety of operations.

Caching

In order to avoid data loss during copying and damage to the file table, the data cache of the recorded file was synchronized using the f_sync function. Given that the disk space could be more than 16 GB, some basic variable types had to be changed to “long log”. FATFS was originally designed with the expectation of embedded devices characterized by limited memory size and low performance. As a result, the read / write functions were performed one cluster discretely per unit of time, which is significantly lower than continuous read / write. A similar problem arose when reading the directory structure when there are more than 100 files in the directory. Reading / writing a single file grew linearly with an increase in the number of files, as shown in the figure. The “x” axis shows the number of files in the directory, and the “y” axis shows the reading time in seconds. Types of curves on the graph for: V2 Sim Fat – disk drive simulator, V1 Fat FAT32 file system for which this problem is missing (not FATFS), V2 Fat64 / 128 FATFS with a disk size of 64 and 128 GB, respectively.



The structure of the cache in this SDK is shown in the figure below. Thus, not only the file structure was cached, but also the sectors for low-level read / write functions of the media device, due to the absence of a hardware cache.

Unique application access to the device

Applications with an integrated SDK should not have access to an external disk device at the same time. The rules to which the application must obey with the SDK used should look like the one shown below in the state diagram.



To establish device access rules, a mechanism based on the Darwin Notification Center was implemented that allows using notifications to allow or block work with the application's device when another application with this SDK is working with the device. When you first start the application, a request is sent to all applications that are subscribed to receive certain notifications. Each application publishes its state for the remaining applications so that the new application can go into an adequate state. In case the device is free, the application changes to the USING state. The PENDING state is intermediate for BUSY and WAIT in case the device is not available or busy. In the event of a communication failure, the state INVALID is used. When the device is finished using the application, it enters the INACTIVE state

Testing the Framework

Writing Unit Tests for all library functions was critical due to the hard deterministic API behavior requirements for end users. Initially, all tests were performed on a disk device simulator. However, the simulator and the library for low-level disk access were written by different people who did not have the possibility of interaction, and therefore the behavior of the simulator did not always coincide with the behavior of the device driver. To this end, a testing scheme on a real device, presented below, was developed. The framework under test was integrated into an application written using the Private API, and installation was performed via Xcode Server Over-the-Air installation . Private API allowed to display the application from the Background of the application using Push Notification and run the necessary tests. The test results were sent via the REST protocol to the statistics server, where they were subsequently presented in the form of graphs and tables. The library also includes the Activity Tracing functions for a more detailed crash log.

Conclusion

Despite the serious flaws in the FATFS, the opensourse library that needs to be improved, another alternative with the ability to use Fat32 and ExFat, as well as an abstract level of support for media drivers, could not be found. If the library is correctly modified with additional caching and buffer read / write functions connected, the functions of copying and reading FAT32 API files are lower than the copy speed of LBA blocks of the hardware driver of the hardware manufacturer by only 10-15%. For newbies in the development and integration of file systems - FATFS can be quite a worthy choice.

useful links

→ FatFs - Generic FAT File System Module
→ Microsoft EFI FAT32 File System Specification
→ Darwin Notification Concepts
→ About Continuous Integration in Xcode