Clobber Builds Part 1 - Missing Dependencies

2014-03-19 by Mike Shal, tagged as make, mozilla, tup

In this series, we're going to look at clobber builds - what they are, why they're needed, and how we can make them a thing of the past. This is part 1 of the series.

Clobber Builds - What are they?

If you do any development for Mozilla, you're probably familiar with the concept of a clobber build. It's not something unique to Mozilla - most Makefiles have a "clean" target, and Visual Studio has a "Clean Solution" option alongside "Build Solution". We also have the CLOBBER file in the root of the mozilla-central tree, which is updated to signal other developers (and release automation) that a particular changeset will need to be clobbered before building properly. A "clobber" just means that any files generated from a previous build are removed, so the next build will start from a clean slate.

Suppose we checkout version A of a project, and then build. We might end up with a set of files, which we'll call A', denoting the built project. We could also do the same thing with version B:

Note that although I used 'hg' in these examples, versions A and B don't have to actually be in revision control. Any changes you make to the project locally could be considered a new "revision", whether it's adding some lines to an existing file, or adding/removing files, etc.

We expect our version control system to always Do The Right Thing, so for example in the following workflow we would be very surprised if we didn't end up at B':

However, with an extra build step in play, we will sometimes not arrive at B':

In this case, some results of the first build are corrupting the second build, so we don't get the result we were expecting. At this point there's no use trusting the build system to get back to B', so it's clobber time!

(From http://mozillamemes.tumblr.com/)

Why You Need to Clobber: Missing Dependencies

One reason why a clobber could be needed is because of missing dependencies. For now we'll stick with some simple build files, but for real-world examples, see bug 890744, bug 984511, or bug 748470. (That last one is actually about removing a FORCE dependency, but removing the FORCE caused things to break because other dependencies were missing). Here's a simple project that compiles a C file, and then writes the output of that program to a file called "output.txt":

$ cat Makefile 
output.txt: main
	./main > output.txt
main: main.c
	gcc main.c -o main
clean:
	rm main output.txt
$ cat main.c
#include <stdio.h>
#include "config.h"

int main(void)
{
	printf("FOO is: %i\n", FOO);
	return 0;
}
$ cat config.h
#define FOO 3

Things work great if we change main.c, but we can quickly see that there's a problem by changing config.h:

$ make
gcc main.c -o main
./main > output.txt
$ cat output.txt 
FOO is: 3
$ echo '#define FOO 4' > config.h 
$ make
make: `output.txt' is up to date.
$ cat output.txt 
FOO is: 3
$ make clean
rm main output.txt
$ make
gcc main.c -o main
./main > output.txt
$ cat output.txt 
FOO is: 4

The highlighted output lines show the problem - in the incremental build case, we ended up with an output.txt file that didn't match the output.txt file from a clean build. From the terminology of the first section, we expected to arrive at B', but we are somewhere else instead.

Although it was easy to identify in this case, sometimes the incremental build failure from a missing dependency can be tricky to unravel. It could manifest itself as a compiler or linker error that shouldn't exist, or a strange test failure that goes away after clobbering. Even more frustrating, the missing dependency could cause things to appear to compile and pass all the tests, whereas the same code would fail from a clean build.

Your example sucks! If you used gcc -MMD you wouldn't have these problems!

Those familiar with make and gcc are probably wondering about gcc -MMD. For those not familiar with it, it is a flag that can be specified on the command-line for gcc to create a dependency file that automatically lists all of the headers used during compilation, so that the developer doesn't have to manually list them all in the Makefile. If I used it in this Makefile, gcc would have created a file like this:

$ cat main.d
main: main.c config.h

Including that in the Makefile would give it perfect dependencies, or so the theory goes. There's just two problems with this:

gcc -MMD doesn't consider dependencies on files that don't exist yet. While correct most of the time, this doesn't account for the fact that a header could be moved earlier in the include path.
Only gcc has -MMD. Makefiles run more than just gcc - there's also shell scripts, python scripts, compiled code that generates other files, etc.

Dependencies on files that don't exist

Let's slightly rewrite the earlier example by moving config.h into a subdirectory. We'll have two directories in the include path, and use -MMD to automatically generate the dependencies file.

$ cat Makefile 
output.txt: main
	./main > output.txt
main: main.c
	gcc -MMD -Ipublic -Iprivate main.c -o main
-include main.d
clean:
	rm main output.txt main.d
$ cat main.c
#include <stdio.h>
#include "config.h"

int main(void)
{
	printf("FOO is: %i\n", FOO);
	return 0;
}
$ cat private/config.h 
#define FOO 4
$ ls public/

This time we have the benefit of gcc -MMD to get the perfect set of dependencies. Unfortunately if the change from version A to version B adds the file public/config.h, we'll still end up in the wrong build state. Only a clobber build fixes the problem:

$ make
gcc -MMD -Ipublic -Iprivate main.c -o main
./main > output.txt
$ cat output.txt 
FOO is: 4
$ cat main.d
main: main.c private/config.h
$ echo '#define FOO 5' > public/config.h
$ make
make: `output.txt' is up to date.
$ cat output.txt 
FOO is: 4
$ make clean
rm main output.txt main.d
$ make
gcc -MMD -Ipublic -Iprivate main.c -o main
./main > output.txt
$ cat output.txt 
FOO is: 5

Admittedly this is a fairly rare case - we need to have a file of the same name as one that already exists, and put it earlier in the include path. However, it does highlight the fact that even the golden standard in make is not perfect. When this does happen, you'll need to clobber.

Everything else in the build system that isn't gcc

Of course, there's more to building software than just a compiler. In the mozilla-central tree for a Linux build, we also run a few bash scripts, the archiver (ar), a linker (eg: gold), compiled C/C++ programs (nsinstall, host_jskwgen, host_ListCSSProperties), yasm, a perl script, and of course, lots and lots of python scripts for IPDL, IDL, WebIDL, Jar manifests, and more. None of these has an MMD flag, so all dependencies must be tracked manually (though we do have some support for automatic python module dependencies from bug 904743).

Any time we change any of these parts of the build system, such as by adding a linker script to an ld command, or tweaking a python script to read from a new input file, we run the risk of forgetting to update the dependencies in the Makefile. Make doesn't report these omissions, but instead assumes that the infallible developer listed all the dependencies correctly. This is a terrible assumption to have -- we all make mistakes, after all -- and it results in an endless game of whack-a-mole as we change things in the tree, introduce new dependency errors, and then have to spend hours tracking them down. In the mean time, every developer and build machine who gets a new CLOBBER file must start over from scratch, often rebuilding the same things over and over again.

Where Do We Go From Here?

In future posts, I'd like to tackle the following topics:

How to fix missing dependency issues once and for all
Why you need to clobber: build configuration changes
How to handle build configuration changes correctly

My hope is that one day we can remove the CLOBBER file from the tree, since it will no longer be needed. Clobber builds will be a thing of the past, and we can instead rely on the build system to do the right thing.

Mike Shal's Blog