Yabs - Yet Another Build System

1 Contents

2 Overview of Yabs

Yabs is a build system that takes the form of a Python library.

Build systems that use Yabs are specified as a Python programme that makes calls to Yabs functions to register rules for object files, executables and other build targets. A primary aim of Yabs is to provide high-level functionality so that these programmes are shorter and easier to write than makefiles or similar.

Yabs consists of the module yabs which contains the core functionality, the module yabs2 containing a command-line interface and support for Make-style pattern rules, and the module yabs3 which implements a particular way of specifying and building executables and libraries.

The primary documentation for Yabs is the documentation strings in the above Python files. This readme is only intended to give an overview of Yabs.

The core of Yabs, defined in the yabs module, is a dependency tree, allowing rules to be defined that make target files from prerequisite files. These rules are expressed as Python functions which take a target filename and, if the rule can generate this target, return a Python tuple containing the command(s) that should be run (or a python function to be called) to create the target and a list of the target's prerequisites required by these commands. This design means that the full power of the Python language is available to the user when writing rules.

For each target that it is asked to build, Yabs calls each rule-function that it knows about, and if the rule-function succeeds and returns such a tuple, it calls itself recursively for each prerequisite. Finally, if the target doesn't already exist, or any of these prerequisites are newer than the target, it calls the command(s) returned in the tuple.

A simple rule could look like this:

def myrule( target, state):
    if target!='foo':   return
    return 'echo hello world > ' + target,
yabs.add_rule( myrule)

This basic functionality can be used by wrapper libraries to support making various builds of object files and executables; see yabs3 for an example of this.

Yabs supports automatic remaking of targets if the command that generates the target is changed, even if the existing target file is newer than all of the prerequisites. See yabs.add_rule()'s autocmds parameter.

On Unix platforms, Yabs can automatically detect hidden dependencies - it detects which files are opened by commands while the commands are running, and uses this information in subsequent builds to force rebuilds of targets when any of these files have been modified. For example, this can provide automatic recompilation of C source files when headers that they directly or indirectly include, are modified. See yabs.add_rule()'s autodeps parameter.

A Yabs programme can be run on its own or imported, without modification, by a different Yabs programme. In the latter case, the rules in the imported programme will become part of the master programme's rules and will work correctly even if the different Yabs programmes are in different directories.

Yabs requires python-2.2 or later.

2.1 High-level example using yabs3

Here's a Yabs script that uses yabs3 to make a rule for building an executable from main.c and other.c:

import yabs.yabs2, yabs.yabs3
yabs3.add_exe( 'myprog', 'main.c other.c')
yabs2.appmakeexit()

If this script is called make.py, then one can build the executable with:

./make.py myprog.exe
All dependencies will be automatically detected - e.g. changes to header files or the compiler flags will automatically be detected and force the appropriate rebuilds.

There are few useful things that yabs3 supports:

All the above commands will default to using debug flags. One can change this with the -b flag: -b gcc,release will default to release flags.

Alternatively, one can specify the build-type as part of the target:

3 Download, history, licence etc

If you are reading this on the Yabs website, then this release is available as the file: yabs-24.tar.gz.

Version: 24.

Date: 2012 December 01.

Yabs is Copyright 2004-2010 Julian Smith.

License: Yabs is released under the GNU Public License. Please see http://www.gnu.org/copyleft/gpl.html, or write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Contact: Julian Smith: jules@op59.net

The main files in this release are:

4 Module yabs.py - core functionality

(For more details, see yabs.html and yabs.py)

4.1 Global state

The yabs module has a global object, yabs.default_state, which contains a list of all registered rule-functions, plus caches for information such as file modification times. Most yabs functions require this object to be passed as a parameter, though the main functions default to yabs.default_state if it is not specified.

4.2 Rule-functions and the make algorithm

Each rule-function has to be registered by passing it to yabs.add_rule().

The function yabs.make() takes a single target and calls each of the available rule-functions until it finds one that claims to be able to make the target. It then calls itself recursively for each of the prerequisites that this rule-function specified. Finally, if any of these prerequisites are now newer than the target, it runs the command(s) return by the rule-function.

If one of the prerequisites cannot be made, yabs.make() carries on calling the available rule-functions until one succeeds. This means that a failed rule may have caused some of its prerequisites to have been re-made successfully. This behaviour is different from traditional Make, which generates a complete tree of rules before running them. The advantage of Yab's system is that it enables rules to use files that are not explicitly represented in the dependency tree, for example one rule could extract an initially unknown number of files from a tar archives and a later rule can use these files.

There is a special last-resort rule, yabs._file_rule() that is always registered, which simply looks for the existence of the target file on disc. This rule is only attempted if no other rules have failed to create the target due to failed prerequisites. This avoids problems when an old version of a target exists, but the standard rules for creating the target fail, e.g. because a prerequisite is incorrectly specified.

4.3 Caching

The yabs module uses several caches to avoid repeating work. These caches are python dictionaries; they exist for the lifetime of the python programme that calls Yabs.

There is a cache of file modification dates, so that finding this information repeatedly for the same file doesn't involve calling the filesystem each time.

Whenever yabs investigates how to remake a particular target, the results of this search are also cached.

Whenever yabs actually remakes a target, it remembers whether the target was updated by the rule. This information can then be used if the same target is required later on - e.g. if a target is a prerequisite of more than one other target.

4.4 Rule details

Rules can be specified as phony. A phony rule isn't required to remake or even create its target, and any file of the same name is ignored.. Phony rules can be useful to give a simple target-name to a real target, such as test. For non-phony rules, Yabs will give an error if a target file doesn't exist after the command has returned.

Rules return a command to run, optionally followed by a list of prerequisites, optionally followed by a list of semi-prerequisites. Other information can also be returned - see the doc strings in the Yabs source for details.

Semi-prerequisites are useful to model things like header files. Yabs attempts to make semi-prerequisites in the same way that is attempts to make prerequisites, but a failure to make a semi-prerequisite is not regarded as an error - it just causes the target to be re-made.

The lists of prerequisites and semi-prerequisites can instead each be a function that returns a list, which can save calculating the list in some cases where it is not needed.

Similarly, the returned command can be either a string or a function that is called to remake the target. In the latter case, the function can also return a string that is run after the function returns. However the string is generated, each line in the string is taken as a separate command and any line that starts with a `-' results in any error from that line being ignored (this is similar to same scheme used by GNU Make (see References).

4.5 Relative/absolute pathnames in rules

By default, all targets passed to rules are absolute pathnames, and the prerequisites and semi-prerequisites returned from rules are required to be absolute pathnames also, so that they work regardless of the current directory in which the Yabs programme is run.

However, a root directory can be specified when a rule-function is registered with yabs.add_rule(). When a target is within such a rule-function's root directory, only the relative path is passed to the rule-function, and the rule-function can return relative prerequisite and semi-prerequisite filenames - they are immediately converted into absolute filenames using the root directory. The rule's command(s) are run with the working directory set to the specified root by prefixing each command with cd <root> && (this works on both Windows and Unix).

Some high-level Yabs rules (e.g. yabs3.add_exe() and yabs2.add_patternrule()) automatically look at their caller's module's containing directory and use this as a root directory. This allows high-level rules, targets and prerequisites to be specified as relative to their caller's module's directory (which is useful for things like executables and phony targets), while still allowing a master Yabs programme to import the modules that specify these targets and use their rules without having to worry about absolute/relative pathname issues.

An example of this can be seen in the make.py Yabs programme, which imports the example-1/make.py Yabs programme, and can be told to make the executable specified in the imported file with (for example):

./make.py example-1/foo.exe

One can use the -W flag to mark certain files as being new, and this will effect how targets in example-1 are built:

./make.py example-1/foo.exe -W example-1/main.cpp

- will do the same thing as:

cd example-1
./make.py foo.exe -W main.cpp

4.6 Rebuilding when commands change

Yabs contains support for forcing targets to be re-made when the commands for these targets are changed (because the build script was changed), even if the existing target is newer than all of its prerequisites.

To use this support, specify an autocmds parameter when calling yabs.add_rule() or yabs2.add_patternrule(). The commands used to create target will then be written to the file target<autocmds>, and Yabs will ensure that this information is used to rebuild targets when the rule is changed.

4.7 Automatic dependencies

Under Unix, Yabs can detect all the files are used by the commands that rebuild a target, and force a rebuild of the target if any of these files are changed. This is done by specifying the autodeps parameter to yabs.add_rule(). The extra dependencies are saved to a filename formed by appending the autodeps parameter to the target filename, and the contents are used as semi-prerequisites in subsequent builds.

For example, header files opened by a compiler will be picked up by the autodeps system, removing the need to use things like gcc -MM. Any similar use of files by other tools will also be picked up, without Yabs having to know anything about these tools.

Commands from an autodeps rule are run with $LD_PRELOAD set to use a special shared-library in the Yabs directory. This shared-library is added to list of prerequisites returned by autodeps rules, and is built from the file autodeps.c by the yabs._autodeps_rule().

At the time of writing, this system is only tested on OpenBSD and Linux. Other Unix systems will probably work fine with some tweaking, but I don't know how to do a Win32 implementation.

Failed attempts to open files are also logged. This enables Yabs to detect a need to rebuild files in certain subtle situations - see the docs for yabs.add_rule(). This works on OpenBSD - see test14. Unfortunately, on Linux the $LD_PRELOAD technique with gcc doesn't seem to log failed file opens, possibly because of glib making direct system calls.

4.8 Concurrency

Yabs can run multiple commands at the same time The simple way of doing this is to use -j <N>, which will run up to N commands concurrently. Yabs takes care to only run independent rules concurrently. For example, this can be used to take advantage of multiple CPU cores.

A more advanced form of concurrency is available: rules can return a resource to Yabs; the resource must provide tryacquire() and release() methods. Yabs only runs the rule's command after tryacquire() returns True. If multiple rules return the same resource, the resource can restrict how many rule commands are run. For example, this can be useful when writing test frameworks that run test on multiple machines - it is easy to write a resource that allows only one test to run on each test machine.

5 Module yabs2.py - useful utilities

(For more details, see yabs2.html and yabs2.py)

The yabs2 module contains various utility functions for generating string representations of the build environment, manipulating filenames etc.

The function yabs2.add_patternrule() allows rules to be specified in a similar way to GNU make rules.

There is also the function yabs2.appmakeexit(), which can be used as the final call in a Yabs programme. It parses the command line parameters for various flags and targets (using similar options to GNU make), and then attempts to remake each target. Finally it exits with an appropriate error code. The parameters are modelled on GNU Make's interface:

Yabs options are:

    -b <build>
            Sets default build string.
    -B      Prints default build strings.
    --changed <file>
            Assume <file> has been changed by a yabs rule.
    --cwd   Print filenames relative to current directory.
    -d      Increment debugging level.
    --d<level>
            Set debug level.
    -e <TRCOE>
            Detailed control of diagnostics when -s and -S are not enough.
            The five sub-params are:
                T:  target
                R:  rule
                C:  command
                O:  output
                E:  exception
            Each of the sub-params should be 0, 1, 2 or 3:
                0: never show
                1: show with: --summary auto
                2: show if rule fails
                3: show when running rule.
            Thus `-e 31222' will always display targets when they are made;
            if an error occurs, the command, output and any exception will be
            shown. If `--summary auto' is specified, the rule is displayed.
            Using E=3 is not useful because exceptions are errors.
    --echo-prefix <prefix>
            Sets prefix used for all command output.
    -h      Show this help.
    --help  Show this help.
    -j <mt> Control concurrency. If <mt> is 0, no concurrency.
            Otherwise build targets as allowed by the resources returned
            by rules. <mt> also governs the maximum concurrency of the
            default resource.
    -k      Keep going as much as possible after errors.
    -K      Equivalent to: -k -e 30222. Useful when one wants to carry on
            after errors, but see information only about errors.
    -l <load-average>
            Only used with -j. Only start making new targets if the system's
            load average is less than <load-average> (interpreted as a
            floating-point number).
    -n      Don't run commands.
    --new <file>
            Assume <file> is infinitely new. Same as -W.
    -o <file>
            Assume <file> is infinitely old.
    -O <file>
            Assume <file> doesn't exist.
    --oldprefix <prefix>
            Assume that all filesnames starting with <prefix> are infinitely
            old.
    --periodic-debug <secs>
            Output brief status information every <secs> seconds.
    --prefix <text>
            Sets prefix used for all yabs diagnostics.
    --prefix-id
            Sets prefix to <user>@<hostname>:
    --psyco
            Attempt to use psyco.full() to optimise Yabs execution.
    --pty   Use pty.fork and os.exec to run commands. This appears to
            work fine on linux, working with commands that read from
            stdin, while still allowing control of echoing and capture
            of the output.
    -s      Quiet operation: don't show commands unless they fail. Equivalent
            to `-e 30233'.
    -S      Very quiet operation: don't show commands or output unless the
            command fails. Equivalent to `-e 30222'.
    --setbuf <stdout> <stderr>
            Sets buffering to use for stdout and stderr. -ve sets to
            system default, 0 is no buffering, 1 is line buffering,
            other values are buffer size.
    --statefile <file>
            Writes current blocking operation to <file>.
    --summary <format>
    --summaryf <format> [+]<filename>
            Adds a summary of failures. <format> is comma-separated list
            of items:
                'rule':     show the rule that failed.
                'command':  show command that failed.
                'output':   show output from rule that failed.
                'except':   show exception from failed rule.
                '':         ignored
                '.':        ignored
                'auto':
                    show just the information that was not output while
                    running the commands - (e.g. if -s is specified, the
                    summary shows the output from commands). This is the
                    default if --summary is not specified.
            --summaryf writes the summary information to the specified
            file. If <filename> is prefixed with a `+', the data is
            appended to the file.
    --system
            Use os.system to run commands. This works better than
            the default when commands use stdin, but doesn't support
            non-echoing or capture of the output.
    --targets
            List info about available targets. Usually this is a list of
            regexes for targets that are defined in terms of a regex.
    --test-flock
            Runs a test that Yabs' flock abstraction works.
    --unchanged <file>
            Assume <file> has been left unchanged by a yabs rule.
    -v      Print version.
    -W <file>
            Assume <file> is infinitely new. Same as --new.
    --xtermtitle
            Writes <user>@<host>:<current target> into xterm titlebar.

Multiple single-hypen options can be grouped together, e.g. `-dds' is
equivalent to `-d -d -s'.

Option values can also be specified with '=', e.g. '-j=7' is equivalent to
'-j 7'.

Anything that doesn't start with `-' is taken as a target.

One useful feature is the -s flag. This turns off display of the command that is being run, but the command is displayed if it fails. Thus you can get very clean output for the common case where things work, but still get detailed information about any build failures.

The -S flag extends this to also hide the output from commands, storing the output internally and displaying it only if the command fails.

yabs2.appmakeexit() also sets things up so that Yabs will output its current state (the list of prerequisite targets that are currently being considered), if it receives a SIGHUP.

6 Module yabs3.py - a particular high-level build system

(For more details, see yabs3.html and yabs3.py)

The yabs3 module supports a particular way of building projects, in which all generated files such as object file are placed near to the source files from which they were compiled, and their filenames encode the parameters and the environment that were used to create them.

It should be understood that yabs3 is not the only way of supporting high-level build targets. Yabs is flexible enough to support many different approaches.

Currently, yabs3 supports building with gcc and, less well tested, Microsoft VC++ under Cygwin.

6.1 Executables

A Yabs programme that uses yabs3 can specify executables in a very simple way, while allowing different builds of these executables to be built from the same source code without a separate configuration command. For example, an executable named foo, built from source files main.cpp and bar.c could be specified with a programme build.py:

#!/usr/bin/env python
import yabs2, yabs3
yabs3.add_exe( 'foo', 'main.cpp somedir/bar.c ../shared/foo.c')
yabs2.appmakeexit()

All object files and executables generated by yabs3 are placed in a _yabs sub-directory.

Building the executable can be done with the following command (assuming build.py is in the current directory and is executable):

./build.py foo.exe

This will create a default-build of the executable. The default build defaults to ,gcc,debug, so the above command will create an executable called something like:

_yabs/foo,debug,gcc2.95.3,os=OpenBSD,cpu=i386,osv=3.3.exe

The default build can be modified using the -b flag or, perhaps better, by inserting flags into the target name:

./build.py foo.exe -b gcc,release,threads
./build.py foo,release,threads.exe

- will create an executable using gcc with optimisation turned on and with support for threading, called something like:

_yabs/foo,gcc2.95.3,release,threads,os=OpenBSD,cpu=i386,osv=3.3.exe

The actual parameters used when calling gcc/g++ are specified in the yabs3.py file itself - the function compile_rule() handles compilation/preprocessing of C and C++ files, while the nested function add_exe.add_exe_rule() handles linking. They both look at the flags implied in the filename that they have been asked to build, and modify the command they run accordingly. The extra OS information embedded in the filename is also used to tune the parameters - for example OpenBSD's gcc uses -pthread to turn on support for threading, but other operating systems' gcc compilers use -D_REENTRANT.

yabs3 contains a particular choice of build parameters (for example, gcc,release is translated to gcc -O2). It is intended that a customised version of yabs3 should be used for a particular site.

6.2 Generated files

All files generated by yabs3, such as object files, are placed in a _yabs/ subdirectory. For object files, this subdirectory is next to the relevant source file. For executables, the subdirectory is in the directory that contains the Yabs programme. In the above example the object files will be called:

./_yabs/main.cpp,debug,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.o
./somedir/_yabs/bar.c,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.o
../shared/_yabs/foo.c,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.o

Putting generated files near to their source files in the _yabs/ subdirectory has the advantage that different Yabs programmes can refer to the same source files, and share the resulting generated files, as long as they import the same yabs3 module so that they all run the same build commands.

It would be straightforward to write an alternative to yabs3 that put generated file in a top-level build directory.

6.3 Preprocessing source files

One can get a preprocessed version of a source file by appending the build and then repeating the suffix:

./build.py main.cpp,debug,gcc.cpp

The preprocessing rule uses some post-processing calls to sed to work around problems with various versions of gcc, and ensure that the resulting file could be compiled without errors.

6.4 Using default builds to simplify target names

The handling of default builds, described earlier for building executables, is done by yabs3 adding a generic phony rule that takes targets and makes a prerequisite by inserting the default build string plus a representation of the environment into the target.

As well as building executables, this phony rule can also be used to easily preprocess and compile source files, using the default build.

To preprocess a file, simply repeat the suffix. For example, to preprocess foo.cpp, do:

./make.py foo.cpp.cpp

This will generate a preprocessed file called something like: _yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.cpp

Similarly, appending .o to a filename will compile it:

./make.py foo.cpp.o

- will generate something like _yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.o.

One can preprocess and compile a file by concatenating the suffixes:

./make.py foo.cpp.cpp.o

- will generate something like _yabs/_yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.o.

In theory, one can preprocess a file more than once, but the resulting filenames can be too long for some file systems.

6.5 Optimised rebuilding when compilation commands change

yabs3's compilation rule uses a modification of Yabs's autocmds technique for ensuring that object files are rebuilt when their compilation commands change.

The autocmds technique would create an extra foo<build>.o.cmd file for every object file, each of which would have to be read, which could take significant time in large projects, so instead, for each directory, yabs3 writes a generic compilation command into the file foo/bar/_yabs/.cpp.<build>.o, only updating this file when its contents would change. The file is then specified as a prerequisite of all foo/bar/_yabs/*.cpp.<build>.o files.

This is a compromise between storing the command that builds each generated file alongside each generated files (which ensures that no files are recompiled unnecessarily, but could be slow), and storing the generic command in some central place, such as /var/yabs/. It has the useful property that restoring a rule to its original form after making a temporary change, will only force a rebuild of targets that are in the same directory as any targets built with the temporary rule.

7 Examples and tests

A simple Yabs build file with hand-written rules could look like:

#! /usr/bin/env python
import sys, os, string
# Add yabs's location to Python's import path:
sys.path.insert( 0, '~/yabs-7')
import yabs, yabs2

# Use a pattern rule for compiling:
yabs2.add_patternrule( '%.o', '%1.c', 'gcc -c -o $@ $<', autocmds='.cmds')

# Use a hand-written rule for linking:
def linkrule( target, state):
    if target != 'myprog':  return None
    prereqs = [ 'foo.o', 'bar.o']
    command = 'gcc -o ' + target + ' ' + string.join( prereqs, ' ')
    semiprereqs = [ 'foo.h']
    return command, prereqs, semiprereqs
yabs.add_rule( linkrule, autocmds='.cmds')

yabs2.appmakeexit()

If the above file is called make.py, then one can build myprof by running: ./make.py myprog. This will output the commands as it runs them, which can be a little noisy. It can be more convenient to use ./make.py -s myprog, which supresses the display of commands.

The above rules use Yab's autocmds support to ensure that targets are rebuilt if the link or compile commands are changed in the make.py file itself.

One can use yabs3 to specify a project more simply:

#! /usr/bin/env python
import sys, os
# Add yabs's location to Python's import path:
sys.path.insert( 0, '~/yabs-7')
import yabs2, yabs3

yabs3.add_exe( 'foo', 'main.cpp foo.cpp')

yabs2.appmakeexit()

The directory example-1 contains a simple Yabs programme example-1/make.py that uses yabs3 to build an executable from two source files. Different versions of the executable can be build with the commands like the following:

cd example-1

./make.py foo.exe
./make.py foo.exe -b gcc,release
./make.py foo.exe -b gcc,threads,release
./make.py foo.exe -b gcc,threads,debug

In the main yabs directory, make.py is a Yabs programme with targets that constitute some regression tests, including checking that the Yabs build inside example-1 deals correctly with a modified header file or a modified command-file. It also has targets that use Yabs to build releases of Yabs itself.

In particular, one can run all the regression tests with the following command:

./make.py test

8 Requirements

Yabs requires Python 2.2 or later. Various versions of Yabs have been tested on various combinations of openbsd-3.6..4.0, a dozen or so Linux distributions, Windows XP, Windows XP/cygwin, python-2.2..2.4, gcc-2.95.3, gcc-3.x, gcc-4.x etc.

9 Background

Yabs' design draws from various sources. Peter Miller's essay Recursive Make Considered Harmful (see References) makes the case for specifying a project in one place, rather than using recursive invocation of a build tool for each sub-directory. Paul D. Smith's extensive website on using GNU make (see References) is required reading for anyone interested in getting GNU Make to do the right thing, particularly the details of auto-dependency generation. The Scons project (see References) is a build system written in python. yabs3's use of filenames to encode build information originates in a GNU Make build system that I wrote a few years ago.

Much has been written about how easy yet powerful Python is to use. Yabs was my first Python project, but even so I think I spent a lot more time working on the design of Yabs than I spent struggling with the language itself. The availability of closures was a delight to someone who had previously only really worked in depth in the C and C++ worlds, and they serve a vital role in simplifying Yabs' code.

IMHO, all build systems are (or should be) basically dependency tree engines. Dependency trees are pretty simple things really, so build systems should in turn be simple. The core function in Yabs is yabs.start_make(); this function scans all available rules, calls itself to build prerequisites, and runs the commands required to build a particular target. It is about 400 lines long.

9.1 Comparison with other build tools

Getting GNU Make to provide the same functionality as Yabs is extremely hard. Much of what Yabs does can just about be done using Make 2.80's $(eval ...) macro, but even then only with a lot of work.

There are also some fundamental limitations with GNU Make that I cannot find a way round. For example, using .d dependency files in a large system with many different available builds, is very slow, because Make has to load all dependency files, not just the ones that are needed for a particular target. Yabs's semi-prerequisites are a much more direct way of getting proper dependency handling without this overhead.

Another example is that I have never been able to tell Make to compile foo/bar/main.c into a file foo/bar/_build/main.o - GNU Make's % pattern matching is simply not powerful enough. Using a full programming language like Python removes this sort of problem at a stroke.

Compared with Scons, I think that Yabs is much smaller and simpler, with no attempt to provide a complete set of rules for all occasions. Yabs is likely to be much faster than Scons on large projects because it uses gcc to find implicit dependencies when it compiles files, rather than using a separate tool; this also guarantees that the implicit dependencies are correct - e.g. there is no risk of the include path being subtley different. Also, Yabs will only read dependency files when necessary. Yabs uses timestamps rather than MD5 checksums when comparing files, although optionally using checksums is on the to-do list.

With Yabs, decisions about what flags to pass to a command are made by Python code in the rule-function itself; there is no use of variables like $CCFLAGS, which are inherently less flexible than code. It also enables Yabs to build different targets using different compilers in the same build, which could be useful when building for a different machine requires some tools to be built and run on the build machine. Similarly, no variable substitution is performed on commands - rules are expected to insert the target/prerequisite filenames in the appropriate places themselves. Yabs does not impose any fixed interpretation such as <dir>/<basename>.<suffix> on filenames - rules-functions can analyse filenames in any way they want, for example using regular expressions. I think requiring the user to write code to make decisions is better than getting them to set particular variables, because it is more general, and because the user is probably a programmer anyway. Additionally, python makes writing small functions very easy.

9.2 Design decisions

Yabs only deals with dependency trees and deliberately has no special handling of compilers, or environment variables such $CFLAGS. Similarly, it has no special knowledge about version control systems. The presumption is that most build problems can be expressed as a dependency tree, and don't need the build system to have hard-coded information about source code repositories etc.

There are some issues of style that are questionable, such as the use of yabs.default_state to encode all rules and state and the partitioning of Yabs into the yabs, yabs2 and yabs3 modules. 'm sure that many won't like yabs3's approach of embedding build information in filenames (e.g. this information could be put in the name of a top-level directory), although remember that yabs3 is just one way of extending the core Yabs functionality - it would be trivial to write a different extension that behaved differently.

At the moment, some of the Yabs scripts modify sys.path to allow a subsequent import to find a particular file, which seems rather crude. This allows Yabs to be used without it being installed into the standard Python module directory (e.g. it could be included in the release of a project) - and also allows one Yabs programme to import another. There is some discussion on the python newsgroups about allowing absolute imports in future, which may help in this area.

Some things I think are pretty robustly defensible: the use of functions to represent rules, the use of yabs.appmakeexit() to provide an easy command-line interface to build scripts, the definition and use of semi-prerequisites and the ability to build targets using text commands or a python function. The support for rebuilding targets when their rules change is very simple yet appears to work well.

One fundamental issue in yabs3 is the complete absence of any build-flags being generated internally and being passed as Python parameters to rule-functions. Instead, rules always look at a target's filename and extract build flags from this filename - they are never passed build information in any other way. This is important, because it means that the filenames convey all the required information, which makes the whole system much more robust because otherwise it would be easy to let filenames and internal build information become inconsistent. Also, it means that intermediate targets can be built independently (e.g. ./make.py foo.c,gcc,release.o) and will be handled identically as when they are built as part of a higher-level target.

10 Future

10.1 Look at using file hashing as an alternative to date stamps

The Scons build system does this. On first glance, I'd imagine it would makes things terribly slow for a large project because of the need to run something like md5 on every source file in a project, but I'm willing to be shown otherwise.

10.2 Handle rules with multiple targets

It would be straightforward to support rules that generate more than one target - this would require changing rule functions to return an extra list of subsidiary targets. I don't know whether this useful though.

10.3 Mark some rules as final

If a rule fails due to Yabs not being able to build its prerequisites, Yabs will carry on looking for a different matching rule. This can result in many generic rules being reported as failing to make a target. It might be better to be able to tell Yabs that if a rule returns non-None, and one of the prerequisites cannot be made, then no other rule should be tried and the build should fail immediately.

10.4 Extensions to autodeps

It would be straightforward to intercept calls to getenv(), and store the name/returned-value in the autodeps output file (which currently only stores the filenames that were opened). Then Yabs could compare the current enviroment with the values in the autodeps file, forcing a rebuild if there is a difference.

There could be a problem if commands modify environmental variables before calling other commands - Yabs would end up always re-running these commands.

autodeps could be made to also output information on files that the command tried to open, but which failed. For example, if a C source file has #include "foo.h", and is compiled with gcc -I /home/me/includes -I /home/you/includes/, and the initial build is done with the file called /home/me/includes not existing, but a file /home/you/includes/foo.h existing, then adding a file /home/me/includes/foo.h should force a rebuild.

Reverse how commands/prerequisites are handled: rules would not list prerequisites. Instead, their commands are run immediately, but any file operations are trapped, and blocked while Yabs makes the requested file. When the requested file is up to date, the original command is resumed.

10.5 Other

Export yabs2's % wildcard matching system for user to use.

Find a generic way of storing yabs3-style per-directory autocmds information.

11 References

GNU make: http://www.gnu.org/software/make/make.html

Recursive make considered harmful: http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html

Paul D. Smith's GNU make page: http://make.paulandlesley.org

Scons - a Software Construction tool: http://www.scons.org