⬆️ ⬇️

How to create, build, install and use packages with programs and libraries for UNIX-like systems

It will be about programs and libraries for UNIX-like systems, distributed as source code (including as tarballs), usually written in C and C ++ (although the same work order can be applied to software in any language). Many things in this article are written specifically for GNU / Linux, although much of the article can be generalized to other UNIX-like operating systems.



The word “package” I understand in this article is a package with source texts, and not a package of a specific GNU / Linux distribution kit, but simply a package coming from the original authors of the software ( UPD from 2017-02-09: except where it is clear from the context that the word “package” is used in another sense).



In this article I will sort out the following questions:

')



I will analyze only very basic things. Those who are typical members of the free software community who program in C and C ++ for UNIX-like systems are usually already known. How to create tarballs (using the example of naked make) and how to install other people's tarballs. Advanced tips for creating "good" packages I will not give. “Advanced” things can be found in the documentation of the build systems, in the wonderful article “Upstream guide” from Debian (at the end there are a lot of links about creating “good” packages). Much in this article could be done differently, my goal: to give at least one way, not to try to embrace the immense.



I will warn you: the beginning will be quite simple, but closer to the end it will be more interesting.



And one more warning. I wrote this article hastily for one person. And after writing I thought, they say, since I wrote it, I’ll post it on Habr so that it does not disappear. Therefore, the article has shortcomings such as the fuzzy structuring of an article, phrases such as “it does not implement its own implementation, ” and so on. I will not correct this. Maybe sometime in the next life, when the hands reach. The choice was between whether to publish in one way or not at all.



So, we start by creating a package with the program Hello, world. To begin with we will be defined with assembly system.



Actually, the package build itself is usually done using the make program (although this is not the only way). The config for make usually lies in a file called Makefile.



There are several options: just use make or use some high-level build system (usually autotools or cmake) together with make, which will actually generate configs for make.



In our example, we will only use make for simplicity. So, create hello.c:



#include <stdio.h> int main (void) { printf ("Hello, world!\n"); return 0; } 


Now the Makefile:



Attention! Works on GNU / Linux. Work under macOS is not guaranteed! This Makefile is not portable to all UNIX-like systems at all! More will come next.



<------> here means tab character.



 PREFIX=/usr/local CC=cc CFLAGS= all: hello hello: hello.c <------>$(CC) $(CFLAGS) -o hello hello.c install: all <------>mkdir -p $(PREFIX)/bin <------>install hello $(PREFIX)/bin/ 


This is far from the only way to write this Makefile. In general, the purpose of my entire article is to give a certain base. What else can you change in this Makefile, you can then find out from other sources.



Also in this article I assume that you already know very basic things about make. I mean that you know that a Makefile is a description of a dependency tree, and that it is needed to collect exactly those files that need to be built. In this article I will talk about using make when building packages, about make virtual targets such as install and all. I also assume that you already know quite basic things about how to run the compiler from the command line, that is, you know what cc -o hello hello.c .



So now let's parse our Makefile.



To begin with, I will say the following. Suppose a user has downloaded a package. He needs to first build it, that is, get the binaries in the directory with the package itself (or in the case of out of tree build, get them in some other directory, which doesn’t change the essence), and then install, that is, copy the received binaries in the place of their final storage in the system.



That is, look. You are logged in as user. Your home directory is / home / user. You downloaded the package, unpacked it, say in / home / user / Desktop / foo. Then you collected it. Collected in the same folder, / home / user / Desktop / foo. Or, if we are talking about out of tree build, you have created another folder / home / user / Desktop / foo-build and compiled it. Now, after the build, you decide to install it in / usr / local. Here you need root rights. With sudo, you install this program in / usr / local.



A few words about the placement of directories in the system. I will analyze only some directories. More information can be found in the Filesystem Hierarchy Standard (supported by many GNU / Linux distributions) and with the command "man 7 hier" on GNU / Linux and possibly other operating systems. In any case, I will tell you how they were arranged, say, five years ago (i.e. somewhere in 2012), all sorts of new developments such as the recent innovations in Fedora, "let's move everything to / usr" I will not consider.





Now about prefix. When you install a package, you must point it to the so-called prefix, i.e., the directory in which everything will be installed. The bin subdirectory will be created in this directory, the binaries will be installed into it, the lib subdirectory will be created, the binary files of libraries will be installed there, etc.



That is, for example, you install a package, pointing it to prefix / usr / local. Then the binaries will go to / usr / local / bin, the binary library files go to / usr / local / lib, and so on.



Now back to parsing the Makefile. PREFIX - this is the prefix that I just talked about. As a default prefix (the user will be able to redefine it) we have / usr / local specified - a good default choice. Usually, it is always indicated as the default prefix when creating packages (unfortunately, some packages still do not do this, then there will be more). Next is CC. This is the standard name for the make variable in which the compiler is put, in this case cc. cc is in turn the commonly used command to run the default compiler on a given system. This can be gcc or clang. On some systems, the cc command may be missing. CFLAGS is the standard name for a variable with compilation flags.



If you just type make, then the goal that goes to the Makefile first will be executed. The usual practice is to call it all. And such a goal usually collects the entire project, but does not install anything. This goal is usually "virtual", i.e. we do not have a file called all. Since our task is to collect only one hello binary, we simply make all dependent on hello. The following is a description of the purpose of the hello assembly. It could in principle be divided into two stages: the assembly of hello.o from hello.c and the assembly of hello from hello.o. I did not do that for simplicity.



Next is install, install. This is also a virtual goal. It depends on all. This is done in case the user dials “make install” immediately, without “make”. For this purpose, we first create the folder where we will install, i.e., as expected, $ (PREFIX) / bin, and then install into it the install utility.



What does install do? This is almost the same as cp. I don’t know the exact differences myself. To install programs you need to use install, not cp.



The binary is set to $ (PREFIX) / bin, because that's exactly what you need to do. Binary files go to the bin subdirectory in prefix, binary library files go to the lib in PREFIX, and so on.



Here I assume that you have installed the so-called BSD install on your system. In GNU / Linux, it is. In some systems it may not be. Maybe some other install that won't work in this situation. This is what I had in mind when I said that work in different operating systems is not guaranteed.



Here I did not consider DESTDIR, which, by the way, is highly recommended for use in a Makefile. I did not even consider the goal of clean.



Ok, let's create the final tarball now, this is the name of the file with the .tar.gz extension, .tar.xz and so on. Place the files hello.c and Makefile in the hello-1.0 folder (it is common to indicate the version number when creating the tarball). Then set as the current directory the one that contains hello-1.0 and type, for example:



 tar --xz -cf hello-1.0.tar.xz hello-1.0 


This will create an archive, inside which there is a hello-1.0 directory, which contains hello.c and a Makefile. This is the way packages are distributed.



C ++ is a variant of the package. The source will be the same, it will need to be called hello.cpp. The makefile will be:



 PREFIX=/usr/local CXX=c++ CXXFLAGS= all: hello hello: hello.cpp <------>$(CXX) $(CXXFLAGS) -o hello hello.cpp install: all <------>mkdir -p $(PREFIX)/bin <------>install hello $(PREFIX)/bin/ 


Note that the standard variable name for flags to the C ++ compiler is CXXFLAGS, not CPPFLAGS. CPPFLAGS is the variable name for the flags for the preprocessor C.



Now let's analyze how to install packages from sorts (any, programs and libraries). Regardless of the build system. This algorithm will also be suitable for the tarball we just created. Suppose that the package you want to build is also called hello.



The first step: download and go to the folder with sorts. There are two options: download from the version control system (I will analyze for the git example) or download the tarball.



First option. git. Make clone:



 git clone <> 


This will create a hello folder. Without version number, since we downloaded from the version control system. Make cd in the created folder, i.e. cd hello .



The second option. Tarball Download the tarball. Then we type the command, say, tar -xf hello-1.0.tar.xz . This will create a folder, for example, hello-1.0 . Usually, if the package creator did everything correctly, the folder name will contain the version number. Then we do cd in the resulting folder, for example, cd hello-1.0 .



Now you need to collect. We will assume for simplicity that we will not do out of tree build (if the author of the package requires out of tree build, there will usually be written instructions on how to do this). So, we will collect in the same folder, where sortsy. That is, in this folder, in which we have now made a cd.



Further actions depend on the build system selected in the project. But regardless of the build system, we will need to specify prefix during the build process. And usually it will be necessary to indicate it at the assembly stage, and not at the installation stage, because often the prefix is ​​hard-coded inside the binary. And this means that after installing the program in a certain place, you cannot just pick it up and move it.



I will give here exemplary instructions that work in most cases. You will see the exact instructions from the author of the project, there may be different nuances.



Be sure to specify prefix when building (whatever the author wrote in the instructions). If you do not specify, the default will be selected. This is usually / usr / local, and is a fairly good choice. And if not? What if the package author specified some other default prefix? You install it is not clear where. In particular, libqglviewer uses / usr as the default prefix, which is completely wrong (I sent a report to the author). So, absolutely always specify the prefix. Read the instructions that the author points to on his website and figure out where to shove prefix.



So, what could be the assembly system. First, it may just be make. This option with bare make is rare. One of the few packages with naked make - bzip ( www.bzip.org ). In the case of our hello-1.0.tar.xz, which we created, we have exactly this option.



So, to build in the case of a bare make, you need to do this:



 make PREFIX=/--- 


(Specifically, in the case of bzip, it is not necessary to specify PREFIX at the build stage. But theoretically, you can imagine a package that takes PREFIX to the inside of a binary. Therefore, in general, PREFIX is needed.)



The next option is autotools. In this case, collect as follows:



 ./configure --prefix=/--- make 


The next option is cmake. Putting it this way (note the point at the end of the cmake command):



 cmake -DCMAKE_INSTALL_PREFIX=/--- . make 


Where is the dot at the end? The fact is that cmake needs to pass the path to the samples. And since we do not have out of tree build, we collect here. Sortsy are in the same place where we are. Therefore, point, i.e. current directory.



In the case of autotools and cmake, the command that generates the Makefile (i.e. ./configure or cmake) writes the prefix to the configs for make, so in the make command (and in the make install command, which will be discussed later), you do not need to specify prefix .



So, collected one of these ways. What's next? Now you need to install.



In the case of bare make, this is done like this:



 make PREFIX=/--- install 


You will need to specify the same prefix that you specified during the build.



In the case of autotools and cmake like this:



 make install 


If you need sudo to write to prefix, then this command will need to be typed with sudo to install. In general, the assembly is always carried out with the usual rights, but the installation is carried out with the rights that are needed to write to the prefix.



Okay, now let's see what we have done. Suppose that we did not install anything, but the hello-1.0.tar.xz that we created before. Suppose also that the prefix that we specified was / foo. Then the folders / foo, / foo / bin (if they were not there before) and the file / foo / bin / hello will appear in our system. What happened? The PREFIX = / foo variable specified on the command line during the assembly overrides the PREFIX = / usr / local specified in the Makefile. As a result, the mkdir -p and install commands specified in the Makefile become:



 mkdir -p /foo/bin install hello /foo/bin/ 


As a result, the binary is put into / foo / bin.



Now I want to talk a little more about prefixes. What prefixes are there?



Prefix /. Hardly ever you have to choose it. It is used for programs that are critical for the early stages of booting the OS (i.e., critical boot elements are in / bin, / lib, etc.) (however, even if you need to install the program in /, it is first installed in / usr, i.e., assemble and install with the / usr prefix, and then move the necessary to / [i.e., move from / usr / bin to / bin, say], in any case, this is what the Linux From Scratch 7.10 authors do package, say, bash).



Prefix / usr. A standard prefix commonly used for programs installed through the package manager. That is, if you installed the program through a package manager, it behaves as if it was built and installed on your system with the / usr prefix. You cannot install packages with the / usr prefix yourself.



Prefix / usr / local. An excellent prefix for installing programs there yourself. Good for the fact that / usr / local / bin is in the default PATH (at least in Debian). That is, immediately after installing the program, you can simply launch the program by name. Because the binary is in / usr / local / bin, and / usr / local / bin is in PATH. The bad news is that there all the programs are mixed. For example, let's say you installed the foo library, and then the bar library. Both are in this prefix. Then the tree may look like this (in a very simplified form):



 /usr/local/include/foo.h /usr/local/include/bar.h /usr/local/lib/foo.so /usr/local/lib/bar.so 


Do you see? Everything is mixed. There is no single folder that contains “everything related to foo” and another folder that contains “everything related to bar”. (Although it is your business, consider that it is really bad or not). It is clear that the same problem is present with any installation of different packages in one prefix. That is, the prefix / usr suffers from the same: packages are “smeared” on the system (here we are talking about packages delivered via the package manager, i.e., those that actually make up the system). Actually, this is one of the striking differences between most UNIX-like systems from Windows. On Windows, each program is in its own folder in Program Files. In most UNIX-like systems, it is “smeared” on the system. ( UPD from 2017-02-09: in Windows, the programs are in fact also “smeared”, let's say, according to the registry, it's just not so striking.) There are GNU / Linux distributions that “solve” this problem, for example, GoboLinux. There, each package in its directory, as in Windows.



Type prefixes / opt / XXX. The / opt folder is supposed to be used as follows: in it you need to create subdirectories, call them package names and use these subdirectories as prefixes. With this approach, the above / usr / local problem (if considered its problem) disappears. Each package will be installed in its own directory. The above example with foo and bar will look like this (I would advise in the name of the subdirectories in / opt also indicate the version number):



 /opt/foo-1.0/include/foo.h /opt/foo-1.0/lib/foo.so /opt/bar-2.0/include/bar.h /opt/bar-2.0/lib/bar.so 


There is also a lack of such a solution. You will have to add all these countless / opt / foo-1.0 / bin directories (for each package) to your PATH yourself.



Prefixes corresponding to home directories. Ie, say, / home / user. I advise when you want to put "only for yourself", that is, only for one user. Or when there are no root rights. Perhaps your configs supplied with the OS are already configured to put ~ / bin in the PATH, provided that such a directory exists. So PATH will be configured as necessary.



Each prefix can contain its own bin, sbin, lib, include, etc.



So, what to choose from this? If you need to put on the entire system, then I would advise / opt / XXX. I myself usually put it that way.



Now about the assembly, installation of the library and its use. A library is being assembled and put up just like any other package, I have already told it above. So let's go straight to use. Here we installed the library in a certain prefix, say, / foo. Now the header files of this library have appeared in / foo / include, and the library binary files have appeared in / foo / lib (.so is a dynamic library, or .a is static, or both).



Suppose you need to build some ac file with this library. How to do it?



First, at the top of the file you need to write #include, corresponding to the subheading header. Well, you need to collect as follows:



 cc -c -I/foo/include ac cc -L/foo/lib -oa ao -lfoo 


Let's figure it out. To begin with, I will say that I (big English and) and l (small English ale) are two different letters. Do not confuse them in the above commands.



The first command compiles, that is, it creates ao on the basis of ac. The second is a link, that is, the final binary based on ao.



In the first command, we specify -I / foo / include. This is an indication of the folder where you need to look for heders. The path from this option will connect to the file specified in #include. That is, if the command line says -I / foo / include, and the file says #include <foo.h>, then you get /foo/include/foo.h, it will be closed.



Here, the -I / foo / include option does not include the connection itself. It only indicates the folder where you want to search, so #include is also needed. That is, you need and -I / foo / include, and #include, one of them is not enough.



Linking -L / foo / lib is an indication of the folder where you need to look for binary library files, i.e., .so and .a files. -lfoo is an indication that, in fact, link this library to the resulting binary. The library name specified in the -lfoo option will connect to the folder specified in -L / foo / lib and you will get / foo / lib / foo, then .so (or .a) will be added here and optionally the version number will be / foo / lib /foo.so or, say, /foo/lib/foo.so.1. This will be the name of the .so file that will be searched.



As with the compilation (ao from ac), both the -L / foo / lib and -lfoo options are needed. -L / foo / lib tells you where to look. A -lfoo gives the final command to link.



Instead of -lfoo, you can directly write the entire path to the library file that you want to link, for example, /foo/lib/foo.so.1. Then the option -L / foo / lib is not needed. It will turn out like this:



 cc -oa ao /foo/lib/foo.so.1 


The library (be it the full path to the library or an option like -lfoo) must be specified after “its” object files, in this case ao (Maybe not, but it's better to do so just in case.)



You can combine our two teams into one, then there will be something like this:



 cc -I/foo/include -L/foo/lib -oa ac -lfoo 


If the library is installed with the / usr prefix (that is, it is simply installed via the package manager), then the -I and -L options are not needed, consider that you have -I / usr / include and -L / usr / lib has already. The same probably applies to / usr / local.



If there is some kind of library called foo, then it is usually packaged for debianes in packages called libfoo (or libfoo1, libfoo2) and libfoo-dev. libfoo contains .so files, and libfoo-dev contains headers.That is, clever Debian developers are able to make several from one package. On Debian's assembly machines, a package is assembled and packaged in several packages.



If you install libfoo and libfoo-dev on your machine, the result will be as if you yourself compiled the foo package from the source code with the / usr prefix. You will have, say, files on your system:



 /usr/include/foo.h (  libfoo-dev) /usr/lib/foo.so.1 (  libfoo) 


Let me remind you that the list of files in this package can be found on the team, say dpkg -L libfoo. Well, find the package by file with dpkg -S /usr/include/foo.h.

Source: https://habr.com/ru/post/321468/



All Articles