Linux Namespacing Pitfalls

2015-10-12 by Mike Shal, tagged as linux, mozilla, namespace, tup

Linux namespaces are a powerful feature for running processes with various levels of containerization. While working on adding them to the tup build system, I stumbled through some problems along the way. For a rough primer on getting started with user and filesystem namespaces, along with how they're used for dependency detection in tup, read on!

Read more »

Visualizing the Taskcluster Index

2015-09-15 by Mike Shal, tagged as graphing, indexing, mozharness, mozilla, taskcluster

I've been working on trying to organize the Taskcluster index in bug 1133074. The index is analogous to the directory structure that we have on ftp.mozilla.org, but it is much easier to organize things based on how we'd like to access them. In FTP, our upload scripts actually ssh into the ftp server and copy files around into the layout we want. But with Taskcluster indexing, we can just add a new route and now we can organize things by revision, platform, branch, or however we'd like.

However, the sheer amount of things we build still makes this tricky. I pulled down the build logs for all the mozilla-central builds I could find, and ended up with 106 builds from buildbot/mozharness (including nightlies and l10n), and 16 builds from Taskcluster. That's a lot of routes to get organized into a usable hierarchy!

Read more »

Try not. Do, or do not.

2015-05-06 by Mike Shal, tagged as clobber, make, mozilla, try

Despite the existence of hg.mozilla.org/try, I sometimes feel that "there is no try", or at least not a try that I would like. My concern for this post covers three main areas: how the input to try is specified, how the output (failure or success) is determined, and how long it takes to run. I'd like to look at each of these in turn and compare our current setup with an "ideal" try server.

Read more »

Clobber Builds Part 4 - Fixing Other Clobber Causes

2015-03-13 by Mike Shal, tagged as clobber, make, mozilla, tup

In Clobber Builds Part 3, we looked at how changing the build configuration can result in broken builds, even if dependencies are perfect. Here we'll look into addressing these issues in the build system.

Read more »

Combining Nodes in Graphviz

2015-01-22 by Mike Shal, tagged as graphing, graphviz, mozilla, tup

Graphviz is a handy tool for making graphs. The "dot" command in particular is great for drawing dependency graphs, though for many real-world scenarios there are simply too many nodes to generate a useful graph. In this post, we'll look at one strategy for automatically combining similar nodes so that a more understandable dependency structure is revealed.

Read more »

Build System Partial Updates

2014-09-10 by Mike Shal, tagged as make, mozilla, tup

There is a fairly long dev-platform thread about partial updates - specifically, running './mach build <subdirectory>'. In this post, we'll compare how this is handled in make-based systems, as well as in tup.

Read more »

PGO Performance on SeaMicro Build Machines

2014-08-20 by Mike Shal, tagged as mozilla, performance, windows

Let's take a look at why our SeaMicro (sm) build machines perform slower than our iX machines. In particular, the extra time it takes to do non-unified PGO Windows builds can cause timeouts in certain cases (on Aurora we have bug 1047621). Since this was a learning experience for me and I hit a few roadblocks along the way, I thought it might be useful to share the experience of debugging the issue. Read on for more details!

Read more »

Moving Automation Steps in Tree

2014-07-23 by Mike Shal, tagged as make, mozilla

In bug 978211, we're looking to move the logic for the automation build steps from buildbot into mozilla-central. Essentially, we're going to convert this:

buildbot build

Into this:

mozharness build

Read more »

Clobber Builds Part 3 - Other Clobber Causes

2014-06-03 by Mike Shal, tagged as make, mozilla, tup

Part 3 in the clobber build series. Today we'll examine some of the reasons that even a build system with perfect dependencies would still need clobbering.

Read more »

Clobber Builds Part 2 - Fixing Missing Dependencies

2014-05-07 by Mike Shal, tagged as make, mozilla, tup

This is part 2 in the clobber build series. Here we'll look at how to fix the issues stemming from part 1 -- missing dependencies -- once and for all.

Read more »

comments powered by Disqus