On some aspects of the organization of the UNIX / Linux file system

1.0 Introduction

After writing the previous article ( Linux: Installing non-distributive programs with the xstow manager ), I have a dual impression. On the one hand, the article is all right, but on the other hand, reviews have shown that there are some discrepancies in the purpose of the different parts of the UNIX filesystem. It turned out that I gave a hammer to people, gave instructions on how to use the hammer, and did not say which nails and where to hammer with this hammer. I will try to fill this gap. In this article I will try, as far as I can, to tell you how the UNIX file system is organized, why it was done exactly this way, why and how to behave in this system.


')
In the future, as examples of UNIX I will use the Debian / Ubuntu distribution.

As we know from life, our whole life is a solution to the problems that arise before us, the emergence of new problems, including those arising from the solution of previous ones and so on, to infinity ...

2.0 Problem: how to place files for various purposes in the OS

So we formulate the first problem, how to place files in the system? On the one hand, we have an operating system and programs, each of which usually consists of several, sometimes many files. If everything is more or less clear with the base files of the operating system, as a rule they should always be in the same place, the situation is more complicated with program files for which installation / removal is possible. On the one hand, it is good when the files are sorted by purpose, on the other hand, it is convenient when the directory trees of the program files are in the directories of the corresponding programs. Let's look at the pros and cons of these solutions in more detail.

2.1 First option: sort files by packages

The first thing that comes to mind is let's decompose the OS files into packages. Those. for each software package there will be a directory in which there will be subdirectories in which the files will already be decomposed in the way the manufacturer of this program needs.

Pros:

• Each manufacturer lays out the files in its directory as it needs.
• It is easy to install / remove programs, copied or deleted the directory, and everything.

Minuses:

• The operating system searches for executable files or shared libraries by a list that is in the corresponding environment variable. For example, the paths to executable files are listed in the $ PATH environment variable. When there are a lot of packages, these lists become really huge.
• There is another compelling reason for which it is desirable to place files of a similar purpose different in the same places. The fact is that for different reasons, the properties of directories / file systems for files of different purposes should be different. This is necessary, sometimes for security reasons, sometimes for reasons of optimal tuning of the system. in UNIX, it is possible to mount file systems in a directory. What does this give? You can, for example, mount a directory for running binary files in read only mode, and then no malicious program can write anything there, and mount another directory in a mode that prohibits launch, and then no one can run anything from this file system. Files that must be changed (for example, database files) are located on another file system, whose parameters can also be optimally adjusted.

2.2 Second option: sort files by function in the system

Pros:

• You can optimally customize file systems based on file functions.
• You can optimally configure permissions on files and directories and mounted file systems.
• It is possible and convenient to give access to such directories on the network.

Minuses:

• There are a lot of files in common directories for which it is not clear which package they belong to. There is a problem with the removal of packages.

3.0 UNIX-style solution

In UNIX, the location of the second option is selected. Those. files of different programs with the same functionality, are in the same directory. For example, executable files of various user programs are located in the / usr / bin directory .

4.0 Problems arising from a UNIX-style solution, installing software

Problems, of course, stem from the cons of the approach. These are difficulties with installing programs, uninstalling programs, and determining which file belongs to which program. This is almost always a non-trivial task.

5.0 Problem Solving: Package Managers

In the future we will call the package manager MP.

The problem formulated in the previous section in various UNIX / Linux distributions is solved by installing, deleting, and storing information about the installed software on a special program, the package manager. In Debian / Ubuntu, dpkg does this. In fact, this is only one program from a whole set of programs that are related to this. dpkg, this is a backend, there are several frontends, apt-get, aptitude, dselect, synaptic.

What does the MP do? At least in Debian / Ubuntu, he does the following:

1. Installing packages.
2. Removing packages.
3. Dependency tracking (when installing the package together with the package, install other packages necessary for this package)
4. Installing packages over the network from remote repositories.
5. Maintain a local database of installed packages.
6. Maintain a local database available for installation packages including the network.
7. Updating databases when changing available packages and / or their versions.
8. Correct installation of package updates.
9. Upgrade the entire system.

A lot, what do you think? And Debian / Ubuntu MP does it all really well.

The package itself is usually an archive of files, a description of a package with several fields, and scripts that are executed during installation and removal of a package. In Debian / Ubuntu, the package file has a .deb extension.

6.0 Problems with package managers

As you know, nothing happens without problems. The use of MP is also no exception and brings its own problems. Of course, the MP solves much more problems than it brings, but still they need to be known and deal with them. So the problems are:

1. Dependencies This is a plus of using MP on the one hand. On the other hand, dependency tracking is a big job for package maintainers. Debian currently includes 25113 packages. Some packages require mandatory installation of others, or vice versa, there are incompatible packages. The result is a large and complex dependency system. All this stuff needs to be checked.
2. Sometimes incompatible package combinations are found.
3. Sometimes there are all sly ring dependencies.
4. Due to the large number of packages that need to be combined with each other, sometimes the versions of packages included in the distribution kit become obsolete.
5. Often, the distribution kit contains software of the wrong version.
6. Despite such a large number of packages, sometimes we need software, which is generally not in the distribution.
7. We cannot install our software in the directory where the MP manages, otherwise our software may conflict with the software installed by the MP. Simply put, we can wipe the files installed by the MP, or vice versa.

7.0 Solving problems with package managers, hierarchy / usr / local

Recently, almost all the necessary software is available in the distribution in the form of packages. But still, sooner or later, as a rule, each user / administrator is faced with the need to install software that is not included in the distribution. Usually there are such options:

Problem 1.	You need to recompile the package with different settings.
Problem 2.	The necessary software is not in the distribution.
Problem 3.	In the distribution is not the version we want.

How to be? How to do it with the least risk to bring disturbances to the system? There are several options available:

Problem 1.	Solution 1:	Recompile the package with different settings. (This is if you need a package with different settings). Practically there are no dangers, although dependencies may change as compared to a package compiled in a distribution kit. Here already, clear pepper, you are responsible for dependencies.
Problems 2.3	Solution 2.	Build your own package and install into the system.
Problems 1,2,3	Solution 3	Install the software collected from source in the hierarchy / usr / local

With Solution 1, everything seems to be clear, let's take a closer look at Solution 2 and Solution 3 .

Solution 2: Build your own package and install into the system.

Pros:

1. You can use all the features of the MP listed in section 6.0.
2. If the package is intended for installation on several machines, you can use the network capabilities of the MP, create your own repository with your packages and install / update these packages in a standard way for the MP.

Minuses:

1. The use of all or only part of the capabilities of the MP requires additional efforts, sometimes considerable. If they do not use, then what's the point in using the MP?
2. If you don’t check your package for compatibility, then it’s very likely that your package will conflict with other packages, and conflicts may be the most unexpected.
3. With the massive introduction of such an approach (creating your own packages), there is the possibility of the appearance on the Internet of masses of poorly tested and poorly compatible packages, as there was one time with RPM, I don’t know how they are with this now.

Debian Policy ² (Ubuntu as a derivative of Debian is guided by it too) says two things directly:

• Software installed locally is installed into the / usr / local hierarchy.
• Software installed locally should be safe from rewriting when the system is updated (meaning updating via the MP).

Thus, we can conclude that for software that is not in the repository, which is installed locally on this machine, or on a small number of machines, the standard way is to install in / usr / local .

If the software is intended to be distributed and installed on Debian / Ubuntu systems, the best way is to package it in .deb packages. BUT! In this case, you are responsible for the dependencies, conflicts and updates of this package.

8.0 Problems related to the hierarchy / usr / local

1. As you know, any solution brings new problems. Not an exception and the solution outlined in the previous section. Everything is fine, as long as we installed one package in / usr / local , as soon as there are more than one package, there is a badly managed bunch of files with problems. described in section 4.0.
2. Another problem is the problem with the / usr / local / var directory . As is known, the data files of the programs are recorded in the / var directory, which can change during the program operation (logs, PID database files, etc.). Often
   / var is done on a separate partition, and now we have turned out that the editable files are in / usr / local / var .

9.0 Troubleshooting the / usr / local hierarchy, possible workflows

Solution of problem 1. described in my article ⁵ , therefore I will not describe the solution in detail. In short, the programs are each placed in their own hierarchy in the / usr / local / stow directory , and then, the special manager (xstow), places symbolic links to the program files.

Solving problem 2. This problem will have to be solved by hand. If you used the stow program, you can, for example, make a directory with a unique name in the / var hierarchy, for example / var / local_var , and then make references to it. But this should be done only if there really is such a need.

10.0 Hierarchy / opt

In the hierarchy / opt according to Policy ^{2, I} install my software from third-party vendors. They install their software in a directory like / opt / package or / opt / provider .

11.0 User problems, hierarchy / home

Again according to Policy ² , user files are stored in the / home hierarchy, in directories like / home / username . If we talk about installing software, it can be installed here. and how to do this, given at the discretion of the user.

When do you do this?

• The user does not have administrator rights on the computer.
• Temporarily, "to see."
• Own scripts.

I usually use a mixture of file layouts "by directories" and "by packages". For software that runs constantly, as a rule, these are my scripts, I create directories / home / username / bin / home / username / etc / home / username / var . I just compile software that I just want to watch or play in the / home / username / sw / softname directory and just run it from there.

12.0 Other uses of the / home hierarchy

The / home hierarchy is often used to house files related to large tasks running on a computer. For example, a project is being developed with the name projectname . Then they start a user with the name projectname in the system and everything related to it is placed in the / home / projectname directory

13.0 Other OS

Once Vovochkina mom told dad:
"do not you think that our Vovochka already
is it time to talk about sex? "
Dad thought and answered:
"probably you are right, only inconvenient somehow."
Mom: "Well, tell me on the example of butterflies."
Dad called Vovochka and says:
"Little Johnny, remember you and I were in
public house? "
"I remember."
"Well, so: the butterflies are exactly the same."


Joke

Freebsd

In FreeBSD OS, in principle, everything is the same, with a few differences. The base system files are located there as in the Debian / Ubuntu Linux family in the main hierarchy. Binary packages there correspond two concepts, packages and ports.

Packages are about the same as packages in Debian / Ubuntu. As a package manager, it uses a set of commands for working with packages (and ports).

A port is a set of auxiliary files that allows you to compile and install the sources into the system. It allows you to automatically download the source from the media (for example, CDROM) or download from the Internet, compile and install into the system.

Both packages and ports are installed in the / usr / local hierarchy.

Just as in Debian / Ubuntu, there may be a situation where there is not a package or a port for software. Then, of course, you need to install software from source. To do this, you can either make a port for this software, or simply compile and install the sources into the / usr / local hierarchy. Of course, since there is a lot of software installed in FreeBSD, you need to be careful, because the likelihood of conflict due to this increases.

Well, going back to the methods described in article ⁵ , I think that if you do not make your port, to maintain order, you can use the xstow program. Although BSD-ists may have their favorite methods, unknown to me, let knowledgeable comrades correct me if I am mistaken.

Windows family

In the OS of the Windows family, for placing software, as a rule, the approach is applied when each software package has its own hierarchy. Some progress is observed in the direction of the file layout for assignments, but this is hampered by:

- The lack of a single tree by the file system (letters denoting partitions, C :, D: etc.).
- Tradition.
- Compatibility requirements with the developed software.

Conclusion

I hope the article will be useful. In principle, it would be possible in another article to analyze in more detail the purpose of other directories and hierarchies of the UNIX / Linux file system.

List of abbreviations used

.deb	Debian / Ubuntu software package file extension
dpkg	dpkg is software that is the basis of the package management system in Debian.
PID	Process id
RPM	Red Hat Package Manager - Red Hat Package Manager
DB	Database
MP	Package manager
OS	operating system
BY	Software
repositories	Software storage with access using MP
FS	File system

Literature

1. Standard Linux File System Hierarchy (File System Hierarchy Standard)
2. Filesystem Hierarchy Standard (Addition to the Debian Policy)
3. http://en.wikipedia.org/wiki/RPM
4. http://en.wikipedia.org/wiki/Dpkg
5. Linux: Installing non-distributable software with the xstow manager
6. man hier