Explicit Variants

In tup-v0.8, tup changed how it builds variants. This allows variants to work on operating systems without FUSE (eg: Windows), as well as better support external tools like debuggers. Unfortunately this results in a backwards-incompatibility in Tupfiles that were built with variants in prior versions of tup. Users of tup without variants should be unaffected by this change.

In general, Tupfiles that use variants need to do the following:

  1. Always use %f and %o flags when referencing inputs and outputs in the command string of a rule.
  2. For cases where you need tell a program where to find generated files that aren't passed explicitly in the command-line (for example, compiler include paths), use $(TUP_VARIANTDIR) (or in lua, tup.getvariantdir()).
Both of these are covered in more detail in the Explicit Variants section below. Note that the files listed as the inputs and outputs in a rule do not need to use $(TUP_VARIANTDIR) or otherwise reference a build directory.

Background - old FUSE variants

In v0.7.X, tup used a FUSE filesystem overlay to redirect where the output files from a sub-program would actually be written on an underlying filesystem. For example, consider a simple compilation rule:

sub/Tupfile 
: foreach *.c |> gcc -c %f -o %o |> %B.o

Without variants, this rule will compile all *.c files in the directory sub/ and output the *.o files into the same directory. The actual gcc commands look like the following:

gcc -c foo.c -o foo.o
gcc -c bar.c -o bar.o

With v0.7.X variants, the build directory was overlayed on top of the source directory. The FUSE overlay meant that the outputs were written to build/sub/ instead of sub/, but the gcc commands are identical to the non-variant case. This disconnect between what the program (gcc) thought it was writing to and what it was actually writing to caused issues with other external programs like gdb that would look in the wrong place for source & object files.

Explicit Variants (v0.8+)

Now in v0.8, variants are more explicit in terms of how the sub-program sees the inputs and outputs. Tup will automatically translate variant paths at the when parsing Tupfiles, and pass along the correct filenames to the subprocess. However, the Tupfile rules may need to be variant-aware in order to work properly.

Using %f and %o

Without variants, use of %f and %o to reference inputs and outputs is optional. For example, the following two Tupfile fragments are equivalent when building in-tree:

: |> touch foo |> foo
: |> touch %o |> foo

Both of these work fine without a variant, but with explicit variants the first case will fail:

* 0) [build] touch foo
 *** tup messages ***
tup error: File '[...]/foo' was written to, but is not in .tup/db.
You probably should specify it as an output
tup error: Expected to write to file 'foo' from cmd 12 but didn't
 *** Command failed due to errors processing the output dependencies.
 [ ] 100%
 *** tup: 1 job failed.

Using touch %o instead will pass along the correct build/foo path to the touch command:

0) [0.002s] [build] touch build/foo
[ tup ] [0.028s] Updated.

Note that tup will adjust the paths for both inputs an outputs. For example:

: |> touch %o |> foo
: foo |> cat %f > %o |> bar

Without variants, the following commands are run:

touch foo
cat foo > bar

With explicit variants, the commands look like this:

touch build/foo
cat build/foo > build/bar

$(TUP_VARIANTDIR)

In some cases, the rules you write won't reference a generated file directly. For example, a script might generate a header file that later compiler commands will consume. However, only the directory for the header file is referenced on the command-line, which you can specify with $(TUP_VARIANTDIR).

In this example, we have a header file that is generated in the include directory, along with a statically created header file in the same directory. A C file in the src directory includes both of these headers. With explicit variants, we need to pass include flags for both the regular include directory as well as the one inside the variant.

Tuprules.tup 
# Add include path for the auto-generated headers inside the variant.
CFLAGS += -I$(TUP_VARIANTDIR)/include

# Add include path for the statically created headers inside the source tree.
CFLAGS += -I$(TUP_CWD)/include
include/Tupfile 
: |> echo '#define GENERATED' > %o |> generated.h
include/static.h 
#define STATIC 3
src/Tupfile 
include_rules

: foreach *.c | ../include/generated.h |> gcc $(CFLAGS) -c %f -o %o |> %B.o
src/foo.c 
#include "static.h"
#include "generated.h"

Without variants, the following commands are run:

include: echo '#define GENERATED' > generated.h
src: gcc -I../include -I../include -c foo.c -o foo.o

Note the duplicate (but innocuous) include flags. This is because $(TUP_VARIANTDIR) evaluates to the same as $(TUP_CWD) when variants are not in use.

With variants, this is the result:

[build] include: echo '#define GENERATED' > ../build/include/generated.h
[build] src: gcc -I../build/include -I../include -c foo.c -o ../build/src/foo.o

The generated header file ends up in the build/include directory while the statically created file remains in the source tree's include directory. By passing in both include paths we can include both files without any FUSE overlay trickery.