Multiple Directories

Now our program grows beyond the confines of a single directory. Here we'll see how Tupfiles can share variables and :-rules.

$ mkdir tup_test4
$ cd tup_test4
$ touch Tupfile.ini
$ mkdir newmath

Note that in this example, we created an empty Tupfile.ini file at the root of our project. This is an alternative to running tup init to create the tup database. When tup can't find the .tup directory that is created by the init process, it will look for a Tupfile.ini anywhere up the tree and automatically run tup init there for you if it finds one. This can be helpful when cloning your repository for different users or machines to reduce the steps needed to get a new developer setup. Note that Tupfile.ini is separate from the Tupfiles that contain the rules to build your program.

We'll start out with the same program as in the first Tupfile example, but this time we want to put the squaring function in a separate directory so it can be bundled into a library with other related files.

hello.c 
#include <stdio.h>
#include "square.h"

int main(void)
{
	printf("Hi, everybody!\n");
	printf("Five squared is: %i\n", square(5));
	return 0;
}
newmath/square.c 
#include "square.h"

int square(int x)
{
	return x * x;
}
newmath/square.h 
int square(int x);

Now we have to create some Tupfiles in order to build our math library and main program. With multiple directories to deal with, it may seem overwhelming when trying to figure out where you put the Tupfiles, and how they should be written. The basic idea that tup uses is that each directory contains a Tupfile, and each Tupfile is independent. You do not need to have a top-level Tupfile call or otherwise include subdirectories -- just create a Tupfile in each directory.

Still, figuring out what to put in each Tupfile may not be immediately obvious, especially if you don't want to repeat yourself in every Tupfile that needs to compile C files. A good approach is to start at a base level, and build your way up. You don't have to write all the Tupfiles at once and get it right on the first try! Just follow these steps:

  1. Build: Create a Tupfile in a single directory, and get it to produce the output you want for that directory.
  2. Refactor: Clean up the new Tupfile so that it looks nice, and maybe pull variables out into common files so that repetition is mostly avoided.
  3. Repeat: Continue this process until your program is fully built.

Once all the Tupfiles have been created, it is a good time to commit them to your version control and take a break!

Build the library

To walk through the process with our example, let's start with a quick look at what we have so far:

$ ls
hello.c  newmath
$ ls newmath/
square.c  square.h

We have two directories with files that we need to build, so we'll end up with a Tupfile in each one. (Note if we had another directory with some documents or something that doesn't require tup to build, we simply would not put a Tupfile there). Since we know the main program will eventually need to link our newmath library, it is probably easiest to start with just building the library itself. We'll create square/Tupfile and get our newmath library:

newmath/Tupfile 
: foreach *.c |> gcc -Wall -O -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

$ tup
[ tup ] Scanning filesystem...0.006s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/2    ] .
[    2/2    ] newmath
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/2    ] newmath: gcc -Wall -O -c square.c -o square.o
[    2/2    ] newmath: ar crs libnewmath.a square.o
[ tup ] Updated.
$ nm newmath/libnewmath.a
square.o:
00000000 T square

Ok, looks like we got our library built. Now we may want to clean up the Tupfile a bit. For example, it may make sense to pull some of the gcc flags out into a CFLAGS variable. This can be especially helpful if we have a bunch of warning options turned on, or lots of -I flags or the like.

Refactoring the library Tupfile

When making changes to a Tupfile, we can use the tup refactor command to verify that our edits do not actually change the commands in the database. If none of the command-lines have changed, tup will simply re-parse the Tupfile and then quit. Otherwise, we will get an error message about what we changed. Let's give it a shot:

newmath/Tupfile 
CFLAGS += -Wall
CFLAGS += -O
: foreach *.c |> gcc $(CFLAGS)-Wall -O -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

When we tell tup to refactor, you can see it will parse the changed Tupfile and then report that it has "No files to delete" -- this is our key that the refactoring had no unintended side effects.

$ tup refactor
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] newmath
[ tup ] No files to delete.

If we had made some meaningful change, for example by changing -O to -O2, then tup will produce an error message with the refactor command:

newmath/Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

$ tup refactor
[ tup ] Scanning filesystem...0.008s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] newmath
tup refactoring error: Attempting to modify a command string:
Old: 'gcc -Wall -O -c hello.c -o hello.o'
New: 'gcc -Wall -O2 -c hello.c -o hello.o'
tup error: Error parsing Tupfile line 3
  Line was: ': foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o'
 *** tup: 1 job failed.

If we didn't intend for the Tupfile refactoring to change anything, we could keep modifying the Tupfile and running tup refactor until all of the refactoring errors are fixed. Of course, if we *do* actually want to change command-lines (such as by switching to -O2 in this case), then just run tup as normal.

Build the program

With the library ready to go, we can now build our main program. Since each Tupfile is independent, we'll create a whole new Tupfile from scratch in the top-level directory. Of course being a typical programmer, we'll be lazy and just copy the existing Tupfile and touch it up a bit. Don't worry about the mess of redundant information for now - we'll clean it up later.

Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> gcc %f -o %o |> hello

$ tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/2    ] gcc -Wall -O2 -c hello.c -o hello.o
hello.c:2:20: fatal error: square.h: No such file or directory
compilation terminated.
 *** tup errors ***
  *** Command ID=24 failed with return value 1

Oh right, forgot about that header. No problem - we can just add an extra parameter to CFLAGS.

Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
CFLAGS += -Inewmath
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> gcc %f -o %o |> hello

$ tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/2    ] gcc -Wall -O2 -Inewmath -c hello.c -o hello.o
[    2/2    ] gcc hello.o -o hello
hello.o: In function `main':
hello.c:(.text+0x1d): undefined reference to `square'
collect2: ld returned 1 exit status
 *** tup errors ***
 *** Command ID=26 failed with return value 1

Getting closer... I guess we should actually link in that library. We have two options here - we can use gcc's -l and -L flags, like '-lnewmath -Lnewmath' to tell it what and where to link it in. Alternatively, we can just add the library as an input. It's up to you how you want your program structured. Just note that for shared libraries, the method that you use to link can affect that paths searched for libraries at run-time. In our case we have a static library, so it doesn't matter.

Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
CFLAGS += -Inewmath
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello

$ tup
[ tup ] Scanning filesystem...0.007s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/1    ] .
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    1/1    ] gcc hello.o newmath/libnewmath.a -o hello
[ tup ] Updated.
$ ./hello
Hi, everybody!
Five squared is: 25

Finally, the program builds and runs! Now we can go about cleaning everything up to make it more maintainable.

Refactor CFLAGS

Let's take a look at our Tupfiles and see what we have so far:

Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
CFLAGS += -Inewmath
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello
newmath/Tupfile 
CFLAGS += -Wall
CFLAGS += -O2
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

Ouch, we have a lot of redundant information here. Obviously we want to separate that out and have one place where we set warning flags and the like for our program. The way this information can be shared in tup is by using the include directive. In this example, we will actually use its cousin, include_rules.

The usage pattern for include_rules is as follows: in each Tupfile that wants to make use of shared variables and rule definitions, write include_rules at the top of the Tupfile (it can technically go anywhere in the file, but usually putting it at the top is easiest). Then, create a Tuprules.tup at the highest level in the directory hierarchy for where it applies. In this example, we want to create some shared definitions for the whole program, so we'll create Tuprules.tup at the top of our project. If there is a sub-directory that requires extra flags, you could create a Tuprules.tup file in that sub-directory as well. Each Tuprules.tup file only affects the current directory and any below it that use include_rules. For variables that are only intended to affect the current directory, you can just leave them in the Tupfile rather than create a separate Tuprules.tup file. Let's start by moving out the CFLAGS:

Tuprules.tup 
CFLAGS += -Wall
CFLAGS += -O2
Tupfile 
include_rules
CFLAGS += -Wall
CFLAGS += -O2
CFLAGS += -Inewmath
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello
newmath/Tupfile 
include_rules
CFLAGS += -Wall
CFLAGS += -O2
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

$ tup
[ tup ] Scanning filesystem...0.009s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/2    ] newmath
[    2/2    ] .
[ tup ] No files to delete.
[ tup ] No commands to execute.
[ tup ] Updated.

A quick note about the include_rules directive: in this case, this is effectively equivalent to the following Tupfiles:

Tupfile (example) 
include Tuprules.tup
CFLAGS += -Inewmath
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello
newmath/Tupfile (example) 
include ../Tuprules.tup
include Tuprules.tup
: foreach *.c |> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

The difference is that include_rules doesn't care if there is a Tuprules.tup missing somewhere along the hierarchy. Also, if you are several levels deep, you don't have to have a bunch of lines like 'include ../../../../Tuprules.tup' at the top of every Tupfile. The idea here is that the top-level Tuprules.tup file can set up project-wide settings. If necessary, sub-level Tuprules.tup files can override or add settings. Finally, the Tupfile has the settings that apply only to that directory.

Refactor :-rules

For our last refactoring we will try to put the redundant :-rule that compiles all the C files in a common location. We could put the :-rule directly into Tuprules.tup, but that would not give us a chance to override or add new CFLAGS settings in the Tupfile itself (as we would like to add '-Inewmath' in the top-level Tupfile). One way to re-use :-rules is to create !-macros. This is due to the way tup parses Tupfiles line-by-line. The man page has some more info on how to use them, so this will just serve as an example. We add the macro definition to Tuprules.tup, and then we can use it in the Tupfile to make the :-rule much shorter. There is still some redundant information among the Tupfiles, but less than before.

Tuprules.tup 
CFLAGS += -Wall
CFLAGS += -O2

!cc = |> gcc $(CFLAGS) -c %f -o %o |> %B.o
Tupfile 
include_rules
CFLAGS += -Inewmath
: foreach *.c |> !cc |>|> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello
newmath/Tupfile 
include_rules
: foreach *.c |> !cc |>|> gcc $(CFLAGS) -c %f -o %o |> %B.o
: *.o |> ar crs %o %f |> libnewmath.a

$ tup refactor
[ tup ] Scanning filesystem...0.006s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    1/2    ] newmath
[    2/2    ] .
[ tup ] No files to delete.

Note that even though we have $(CFLAGS) when we initialize the !cc macro, it does not actually expand until we use the !cc macro in the :-rule in the Tupfile. This allows us to add the -Inewmath flag in the main Tupfile.

Let's try to make the 'ar' command a !-macro too.

Tuprules.tup 
CFLAGS += -Wall
CFLAGS += -O2

!cc = |> gcc $(CFLAGS) -c %f -o %o |> %B.o
!ar = |> ar crs %o %f |>
newmath/Tupfile 
include_rules
: foreach *.c |> !cc |>
: *.o |> !ar crs %o %f |> libnewmath.a

Notice how the !cc macro specifies the output in the macro, since it is the same for all C files. However, the library can be named anything, so we specify the output in the actual :-rule that uses the !ar macro rather than in the macro definition itself.

Let's take a final look at the Tupfiles we ended up with:

Tuprules.tup 
CFLAGS += -Wall
CFLAGS += -O2

!cc = |> gcc $(CFLAGS) -c %f -o %o |> %B.o
!ar = |> ar crs %o %f |>
Tupfile 
include_rules
CFLAGS += -Inewmath
: foreach *.c |> !cc |>
: *.o newmath/libnewmath.a |> gcc %f -o %o |> hello
newmath/Tupfile 
include_rules
: foreach *.c |> !cc |>
: *.o |> !ar |> libnewmath.a

The syntax may take a while to get used to, but it should be pretty easy for you to add a new flag to CFLAGS, or create another subdirectory to make a new library. You may even find better approaches -- perhaps you will find it easier to set all the variables at the top and put include_rules at the bottom!