One of the main functions of the Mail.Ru Cloud client's desktop client is data synchronization. Its goal is to bring the folder on the PC and its presentation in the Cloud to the same state. In developing this mechanism, we met with some, at first glance, fairly obvious features of various file and operating systems. However, if you don’t know about them, you can face some rather unpleasant consequences (you won’t be able to download or delete the file). In this article, we have collected features, the knowledge of which will allow you to work correctly with the data on the disks and, possibly, will save you from the need for an urgent hotfix.

1. Events from the file system do not guarantee a complete picture of what happened.

Any directory synchronization mechanism requires monitoring changes in the status of files and folders. The benefit of the API of each operating system gives us this opportunity. We use ReadDirectoryChangesW for Windows, FSEventStream for macOS and inotify for Linux. And already there await unpleasant moments. The fact is that under macOS it is impossible to say with certainty what kind of event came from the file system. You can easily get CREATED, DELETED, RENAMED, MODIFIED on a file in one event. And, it seems, everything is logical: if there is a deletion, then the file is no longer there, however:

$ rm 1.txt && echo "hello" > 1.txt 

will come one event:
')

 1.txt: CREATED | REMOVED | MODIFIED 

Therefore, one has to use additional event checking mechanisms to understand exactly what happened to the file or directory.

In inotify, the event queue can overflow , and you can start losing them until you pick up some events from the queue. At the same time, lost events will not be compensated for you in any way, and you will need to perform costly operations such as bypassing the disk.

2. With symbolic links it will not work as with ordinary files.

Symbolic links can be looped: A -> B -> C -> B. You can solve this problem, for example, by using the inode number (a unique file number or folder in the current partition of the disk, but about them just below). In our case, we keep a list of inode of symbolic links, which are passed to the current directory. If the inode of the current symbolic link is the same as the one already in the list, then consider it looped and skip.

The symbolic link may be broken. If at some point the content that the symbolic link points to is moved or deleted, the link will become unavailable. It is important to properly handle this moment.

If you are subscribed to an event from a directory in which you have symbolic links to other directories, then content change events will not come via a symbolic link.

3. File and folder names may be in the wrong UTF-16

There was one interesting bug. In the local tree of the user who reported the problem to us, there was a file. However, when trying to read it, we understood that there is no file. It seems to be a logical situation when at the time of our work the file is deleted. But the next listing of the directory file was again in place. The fact is that under Windows you can create an invalid UTF-16 encoding. More precisely, the name may contain an invalid surrogate pair . Convert this name to UTF-8, and then back to UTF-16 by standard means (WideCharToMultiByte, MultiByteToWideChar) will not work. Take an example:

 wchar_t name[] = { 0xDCA9, 0x2E, 0x74, 0x78, 0x74, 0x00 }; 

Surrogate pairs consist of High and Low values ​​and are needed in order to expand the range of encoded characters. High Surrogates are in the xD800 - xDB7F range. Low Surrogates in the DC00 range - DFFF. In our title, we took High, but did not take Low. So we got invalid UTF-16.

We convert this name to UTF-8, then back:

 wchar_t name2[] = { 0xFFFD, 0x2E, 0x74, 0x78, 0x74, 0x00 }; // " .txt" 

The symbol representing the beginning of the surrogate pair breaks. Call on this name will not work.


We in the synchronization module always work with UTF-8. We receive events or listing from the file system and convert the names to UTF-8. The server also works with UTF-8. When accessing the file system, we will convert UTF-8 back to UTF-16. The problem was solved by the prohibition of synchronization of invalid UTF-16.

4. Pitfalls when working with inodes

For a long time, the desktop client did not support renaming files and folders. Instead, the rename event was handled by deleting files in one place and creating them in another. This mechanism worked quite long and steadily. The fact is that deleting a file from the cloud is only removing the link to this file from the user tree. The file itself remains for some time to live on the server so that you, for example, could later restore it from the recycle bin. Thus, deleting and creating a file in another place was done only by sending meta-information to the server, which simply deleted the link to the file from one place and created it in another, without even opening a local copy. However, with the advent of shared folders, we began to realize that we had to handle just the movement (in order not to lose the shared folder attribute and not disconnect the attached folder).

It's one thing when a renaming event comes from the file system. There are no problems. Tyts-tyts and renamed. And if the application is turned off? We need some information on which we will detect the renaming event. There were several options for detecting movement:

• Compare the hierarchy of files and folders. Very difficult process, even on the basis of the fact that trees are stored in RAM.
• Create hidden files with service information in each folder, by which we will understand where the folder has moved or what it has been renamed to. However, this causes some difficulties, including the fact that the user can change and edit these service files, which can lead to unpleasant consequences. Yes, and "follow" in each directory did not want to.
• Inodes. On this option, we stopped.

Inode - inode. It is denoted by an integer and represents the identifier of a file or folder in a particular file system.



A bit more humane description of "how it works," I recommend reading in this article . In POSIX, we get the inode from stat (st_ino), in Windows, GetFileInformation (nFileIndex). And everything seems to be simple:

1. The client is restarted, we load the cached view of the file hierarchy.
2. We compare with the fact that now lies on the disk in fact.
3. We find nodes whose inode numbers are absent in the place where we assume, but are in some other place.
4. Move these nodes.

However, with inodes you need to be very, very careful. Here are some of the pitfalls we encountered.

4.1. Hardlinks

Each link of this type to one file has the same inode number. We will not detect renaming if there are hardlinks in the tree. Hardlink cannot be created on a folder (or almost impossible), because there are no special problems here.

4.2. Inodes may work differently than you expect.

On some file systems, inode numbers are assigned differently than they should (well, or it seems to us that they should). We assume that their numbers do not change when the file is renamed. We also assume that if the last file on the FS with inode 9 is deleted, the next file will have inode number 10. Unfortunately, some file systems disagree.

Under macOS on FAT , new files are created (not folders) with inode number 9999 ... When you rename these files, the inode number does not change. When editing these files, the numbers change to ordinal values, which we expect to see:

 $ touch 1.txt $ ls -i 999999999 1.txt $ echo "hello" > 1.txt $ ls -i 223 1.txt 

Ext4. The fact is that if on this file system (which is standard in most Linux distributions), delete the file with inode number 9 in one place and create a new file in another place, it will have an inode with the number not 10 or higher, but 9 .

 $ touch 1.txt $ ls -i 270 1.txt $ rm 1.txt && touch 2.txt $ ls -i 270 2.txt 

Those. on this file system, the inode becomes the first free number. It broke our logic a little. The solution came by itself: if the folder is renamed or renamed, we compare the inode numbers for the folders and the hash + size for the files for its content. If directories match by 70% or more, rename it. For files - if the hash + size match.

Given that inodes numbering in different file systems works differently, we have a check to see if inodes work as we expect: when the synchronization module starts, the test behavior is reproduced for testing. If it is what we expect, then you can work with inode numbers. Otherwise, we continue without renaming support.

5. Programs store many service files on disk.

Operating systems and programs of different popularity use service files on the disk, which do not need to be synchronized. Below is a list of files and masks that we thought should be ignored:

Windows:

• desktop.ini - stores user settings for the current directory;
• Thumbs.db - thumbnail caches for images;
• Files starting with "~ $", or ". ~", or starting with "~" and ending with ".tmp" are a fairly common pattern of temporary files. Files of this template also creates Microsoft Office when editing documents.

macOS:

• .DS_Store is an analogue of desktop.ini under Windows;
• Icon \ r - quite an interesting file, when listing the file is displayed as “Icon?”, Stores information about the image on the directory in which it is located;
• files starting with "._" - there were a lot of templates instead, however, various software preferred to use its own temporary file format, after which it was decided to ignore files using this mask.

Linux:

• .directory is an analogue of desktop.ini under Windows and .DS_Store under macOS, relevant for some window managers.

6. Features paths in Windows to files and folders

Paths under Windows, of course, deserve special attention. For paths that exceed the value of MAX_PATH (260 characters), you must use the prefix "\\? \". By the way, this prefix should be used for CreateFile if you are going to open a COM port.

Windows for each file or folder with a name longer than 8 characters creates short aliases (also called “8.3”). Aliases are always in high register, contain the "~" sign, followed by a digit, increasing if such an alias is already taken (for example: "C: \ PROGRA ~ 1 \" ). The content of these signs is necessary, but not enough to understand - before you is the usual name or a short alias. WinApi can turn short paths back to long ( GetFullPathName ). However, it must be remembered that it will not turn the path into a long view if such a file does not already exist.

If someone opens the file using CreateFile using a short path and modifies it, then in the event from the file system (using ReadDirectoryChangesW) you will come to the same short path. In this regard, we try to turn them into long as soon as possible. By the way, you can see the alias, if you enter "dir / x" from the desired directory in the Windows command line.


Another unpleasant feature that cannot be missed: files and folders with a dot at the end cannot be opened using Explorer (true for Windows 7):

7. Conclusion

For each file system, the synchronization algorithm had to be adapted accordingly. The best solution in our case was to recreate the test environment at the start to check the folder that the user chooses. And if the tests do not pass, then we learn a new feature, and the user either forbids working with this folder, or disables some functionality. I hope the features we encountered will help you to avoid difficulties when working with the file system.

If you have questions or comments, feel free to ask them in the comments or write me personally at a.skogorev@corp.mail.ru.