SCons and Waf build systems overview

I am a developer and have been using Python for the last couple of years as my main language. However, from time to time there are tasks when you need to write in C / C ++. There are different systems with which you can build such projects. Classics are make and autotools. I want to focus on alternatives such as SCons and Waf . The goal of the post is not to prove that they are better or worse than make. I just want to make a short digression so that it becomes approximately clear what it is, why it is and how to start working with it.

To make the conversation substantive, I propose to consider the system in practice. I decided to use a simple project, which requires typical, but not always trivial assembly tasks. We will make a simple web-server, the purpose of which is: to produce a static page, which is prepared in a separate html file, but which eventually must be embedded in the executable file. That is, at the assembly stage, the source code with the C code must be compiled using the html code. As the server library, we use mongoose , the source code of which we put inside the project and we will assemble them into a static library, which later we link to the executable file. I think the task is clear.
')


So why are these two systems chosen for experience? After all, there are still Rake , CMake , Ant / NAnt and others. The answer is in the first sentence: they are based on Python, and I know and love it well, so the threshold of entry for me should have been rather low. Let's start ...

Training

In order for the project to be collected via SCons, it must be installed in the system, and the SConsturct script must be in the project root. At least in Ubuntu, the SCons package is present in the standard repository and the installation does not present any difficulties: `apt-get install scons`.

To build a project through Waf, you need Waf itself, which comes in the form of a single file and is placed directly in the project root. That is, it is not necessary to install Waf into the system, and users of the code receive a project from the version control system already with “batteries”. In addition, you need the assembly script itself, which should be called wscript.

The tree of our project looks like this:

~/devel/stupid-server$ tree
.
|-- external
| `-- mongoose
| |-- mongoose.c
| `-- mongoose.h
|-- html
| `-- index.html
|-- html2c
|-- SConscript
|-- SConstruct
|-- src
| `-- main.cpp
|-- waf
`-- wscript


In addition to the above, you can see two more files: html2c and sconscript. I will tell about them further.

Compilation and linking

Scons

Let's postpone the html conversion for now and build the server. Let him give an answer for the time being, which is hard-wired in our main.cpp.

We need the following: the system must compile mongoose into a static library, then compile src / main.cpp and link it all into one executable file. In this case, all the artifacts would be nice to put in a separate directory, so that if something happens, it would be possible to delete it without looking back.

Let's start with the SCons. The SConstruct script for this looks like this:

env = Environment(
CPPPATH = 'external/mongoose',
CFLAGS = '-O2',
)

mongoose = env.StaticLibrary(Glob('external/mongoose/*.c'))
env.Program(Glob('src/*.cpp') + [mongoose], LIBS=['dl', 'pthread'])


Here, the first line we define is Environment - the Centonsbolt of the SCons system, in which we set up the necessary tools. In our case, we add the include path to mongoose and add a flag to build with optimization. The next line we define how to collect mongoose. We say that it is a static library and all .c files in the external / mongoose directory are subject to compilation. In our case, the file is one, but if there were a lot, Glob would spare the need to list them. Finally, in the third line, we define how to build the executable file. We say that it consists of all .cpps in the src directory and a pre-defined mongoose, and in addition it is necessary to link the standard dl and pthread libraries.

I put emphasis on what we define exactly how and what to collect. The build in SCons does not go line by line, as described in the script. Instead, the system uses declarations and determines the sequence itself, taking into account the dependencies between the goals and the possibility of accelerating the assembly in several streams due to the shuffling of tasks.

In general, the description for both systems says that if you know Python, then everything will be simple, because the build scripts are written on it. In practice, this turns out to be cunning, because syntactically it is Python, but for real work you need a clear understanding of the ideology and scheme of operation of these systems, which comes only after intensive smoking of documentation and often the sources of the systems themselves.

Let's go back to the SCons. Now to build the project we need to dial in the terminal `scons` and voila: the web server is assembled! However, in the best traditions of make, build artifacts appear directly next to the source code than clutter the structure. To solve this problem, you will have to create an additional SConscript file to which to transfer the declaration of goals. And from the main SConstruct it is necessary to say: “Collect this SConscript in this directory”. Unfortunately, this is the only way, which however is protected in the documentation as a feature. Our scripts now look like this:

# SConstruct

env = Environment(
CPPPATH = 'external/mongoose',
CFLAGS = '-O2',
)

SConscript('SConscript', variant_dir='build-scons', exports=['env'])

# SConscript

Import('env')
mongoose = env.StaticLibrary(Glob('external/mongoose/*.c'))
env.Program(Glob('src/*.cpp') + [mongoose], LIBS=['dl', 'pthread'])


Now all the guts will be neatly stored in a separate directory build-scons.

We left the definition of Environment in the parent file, since this is good practice: there may be many child scripts, and the Environment they will have in common. The transfer between files is carried out by a plain-curved mechanism: through the exports parameter and the Import construction, respectively.

Waf

Now the same on the waf:

top = '.'
out = 'build-waf'

def set_options(opt):
opt.tool_options('compiler_cc')
opt.tool_options('compiler_cxx')

def configure(conf):
conf.check_tool('compiler_cc')
conf.check_tool('compiler_cxx')
conf.env.append_unique('CCFLAGS', '-O2')
conf.env.append_unique('CXXFLAGS', '-O2')

def build(bld):
bld(
target = 'mongoose',
features = 'cc cstaticlib',
source = bld.path.ant_glob('external/mongoose/**/*.c'),
export_incdirs = 'external/mongoose',
)

bld(
features = 'cxx cprogram',
source = bld.path.ant_glob('src/**/*.cpp'),
target = 'stupid-server',
lib = ['dl', 'pthread'],
uselib_local = ['mongoose'],
)


The first two lines determine what to consider as the root of the project and where to put the artifacts.

As you can see, in Waf this is more logical. Next comes the set_options function. It is called Waf in order to supplement the existing standard command line parameters with user ones. In our case, a standard set of parameters is added in order to affect the operation of the C and C ++ compilers.

Next comes the configure function. It is called when we say `./waf configure` and is intended to set all the environment variables and search for all the tools that will then be used when building with the` ./waf build` command. The division into two stages: configuration and assembly is similar to that used in autotools and serves to accelerate. For everyday use, configuration is rarely required, and the build is constant. This saves a significant amount of time. In our case, in the configuration, we ask you to determine if there are C and C ++ compilers on the machine, remember where they are and then add the optimization flag to their settings.

Next comes the build function, which takes as its parameter a certain build context, the centrobolt of the Waf system. This context can be called as a function, thereby determining how and what to build. We call it twice: to build mongoose and to build the executable file itself. In each of the calls, the source code, the name of the target and a set of features are determined. This kit tells Waf that you need to use tools to turn source into a result. In the first case, we use the C compiler (cc) and the archiver (cstaticlib), in the second - c ++ - the compiler (cxx) and linker (cprogram). In addition, for the executable file with the lib argument, we specify the required standard libraries. And with the help of the uselib_local argument we are talking about linking with our local mongoose.

It is very strange that a bunch of crutches like uselib_local are used to link with their own artifacts in Waf instead of allowing adding target objects to the source list, as is done in SCons. But it is as it is. On the other hand, a nice advantage is that it includes the path to the mongoose encapsulated in the mongoose target itself, and not announced somewhere at the top level. Only targets that depend on mongoose will receive this additional path as a directive to the compiler.

Now, having such a script, it’s enough for the build to call `./waf configure build`. And in consequence, when working on the source code, you can restrict to `./waf build`.

Build html

I don’t know any generally accepted ways of getting a C code from html, so I wrote a tiny html2c script that makes it out of this html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/1999/xhtml">
<html>
<body>
<h1>Hello world from stupid server!</h1>
</body>
</html>


such .c:

// Autogenerated by html2c. Think before edit
const char htmlString[] =
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.1//EN\" \"http://www.w3.org/1999/xhtml\">\n"
"<html>\n"
" <body>\n"
" <h1>Hello world from stupid server!</h1>\n"
" </body>\n"
"</html>\n"
;


To use this line from the main file, use a simple technique:

extern "C" const char htmlString[];


From the point of view of programming, it is not very elegant, but for our purposes it is the most.

Scons

In order for SCons to understand how to get from strange files of one type of others, it is necessary to define the so-called Builder. Thus, our Environment will acquire an additional method like StaticLibrary or Program, but which is able to solve our problem:

# SConstruct
env['BUILDERS']['Html2c'] = Builder(
action = './html2c $SOURCES > $TARGET',
src_suffix = '.html',
suffix = '.c',
)


Now we have env.Html2c, which we will use in our SConscript script:

html_c = env.Html2c('html/index.html')
mongoose = env.StaticLibrary(Glob('external/mongoose/*.c'))
env.Program(Glob('src/*.cpp') + [html_c, mongoose], LIBS=['dl', 'pthread'])


The method returns to us the declaration of the target object, which we then intuitively add to the source code of the main program.

Waf

First, at the configuration stage, we need to find the html2c utility and save the path to it in the environment variable for further use:

def configure(conf):
# ...
conf.find_program('html2c', var='HTML2C', mandatory=True, path_list=[conf.srcdir])


Then you need to define a new transformation, so-called. chain, which tells Waf how to get .c from .html:

import TaskGen
TaskGen.declare_chain(name='html2c', rule='${HTML2C} ${SRC} > ${TGT}', ext_in='.html', ext_out='.c', before='cc')


Here we used the $ {HTML2C} variable, which was previously set in the configuration step.

The nice thing is that Waf does not add any magic to imports, as SCons does. It is always approximately clear where the legs grow from, in this case from the TaskGen module. Therefore, if necessary, you can always just find the place of interest in the source code of the system.

Now that Waf knows how to do the conversion, we can simply add html to the source program's source list:

def build(bld):
#...

bld(
features = 'cc cxx cprogram',
source = bld.path.ant_glob('src/**/*.cpp') + ' html/index.html',
# ...
)


The observant reader might have noticed that for the new transformation we didn’t add anything to the features. This is due to the fact that the `TaskGen.declare_chain` mechanism is a simplified, universal, high-level way to extend Waf. This has a very unpleasant downside: after such a declaration, all the .html that waf sees in the source will be first converted to .c. That is, even if you wish to upload html via ftp to any server for a separate purpose, the system kindly converts them into .c and it will download them.

A complete solution that takes into account the features is surprisingly difficult and requires a deep understanding of Waf. I decided not to bring him here in order not to put you to sleep.

Nice chips

Both systems have built-in scanners of C / C ++ source dependencies. That is, you can forget about the declaration of dependencies on .h files.

Both systems have built-in clean-system. Calling `scons -c` will remove all artifacts, and` ./waf clean` and `./waf distclean` will remove assembly artifacts and configuration step caches files, respectively.

Both systems are cross-platform.

Scons better than waf

SCons has a richer history, has more people in its community, is mentioned in more publications and is used in more projects.

The SCons documentation is one level higher than the Waf documentation. Both systems have e-books, but in the case of Waf, its contents become clear only from the twentieth time.

Scons script ideology is usually more understandable and intuitive. Often, when you are not quite sure how something is being done, you try to do it by analogy and everything turns out. In Waf for many tasks, you need to search for your crutch and deal with its exclusive interface.

In SCons, you can usually do the same thing in one or more similar ways. The flexibility of Waf, which in some places turns into curvature, makes it possible to achieve something in 1000 ways, but only one of them is “correct”, and all the others will come back to glove with further support and work.

Waf is better than SCons

Waf was created relatively recently by a man who stood at the origins of SCons. Therefore, it takes into account and avoided the large conceptual failures of SCons, which in large and complex projects affected productivity, ease of maintenance, cross-platform, etc.

Waf is much faster than SCons. I did not check, but it is argued that on a project with several thousand source files with a large number of dependencies, it surpasses SCons 10-15 times. Thanks to the separate configuration and assembly, as well as a more intelligent system for distributing low-level tasks. By the way, if we compare it with the usual make, then they go nostril to the nostril: Python is slower, but make loses an advantage due to the recursive spawn of the processes of itself in the subdirectories.

The output of Waf to the console is incomparably nicer than the noodles from under SCons and make. It is colorful and visual, and the errors and warnings are very easy for the eyes.

Waf mobile. The whole system is in a single file less than 100Kb in size directly in the code, and therefore allows you to forget about the need to install yourself on the build machine. All you need is Python, which is on most Posix systems.

Waf allows working with source files that are outside the project root directory. For example, with libraries installed in the system, but available only in the form of source code. If a similar problem arises in SCons, it is much easier to go hang yourself.

Both are "good"

Documentation for both systems is not an example. Although for both systems there are e-books that allow you to start working, the usual reference / manual is by and large only available to SCons. And then it represents one huge footcloth, which can only be searched for via Ctrl + F. When expanding Waf, in complex scenarios, when it comes to the aspect-oriented programming that is widespread in it, you almost certainly have to refer to the source code to find the ends. In the documentation, as a rule, there is simply no such information.

The fact that the build scripts are just Python: a duck. In both cases. In view of the specifics, the code is executed not empirically: line by line, but in arbitrary order, as the system decides. Rather, the code, of course, is executed sequentially, but if it says `build_this_thing`, this does not mean that the build will happen now. It only means that there is such a thing, and that's how it should be collected if anything happens. A sort of XML on Python is obtained.

The source code of both systems is also not a fountain. In some places it is written in such a way that if I wrote something similar at work, I would just quit in order to at least somehow make amends to my colleagues. Although the creators can be understood: their native environment is C / C ++ and they may not know most of the details of the Pythonic culture.

Material

All files of the project, disassembled in this post are available in the repository on BitBucket .