Subject: Re: Have pkgsrc building make use of SMP?
To: J Chapman Flack <flack@cs.purdue.edu>
From: Nathan J. Williams <nathanw@wasabisystems.com>
List: tech-pkg
Date: 04/06/2005 13:33:47
J Chapman Flack <flack@cs.purdue.edu> writes:

> On the other hand, "the target-structure of a Makefile contains all the data"
> might be an optimistic statement of the way most Makefiles seem to get
> written.  I'd hazard they often contain any subset of the complete dependency
> data that allows building successfully in the author's environment and
> doesn't get too many complaints from outside.  :/  Parallel building is
> probably a more demanding test case than what's often used.

This is definitely true. A particular source of problems is recursive
make processes invoked to do builds subdirectories; this doesn't
really express a dependancy ordering to the tool. A package that
builds "liba", "libb", and then "program" which links against both of
those only succeeds because liba and libb are first in the list of
subdirectories to process. It was claimed earlier that GNU make seems
to have less of a problem with this; perhaps it is less agressive
about firing off recursive makes in parallel?

In the NetBSD tree we solve this with the .WAIT symbol in our
subdirectory lists, which creates a barrier to parallel execution (and
thus expresses the crude ordering that everything after it depends on
everything before it), but it's far from ideal. Ideal, unfortunately,
means rewriting everything to never invoke make recursively, so that
the entire dependancy tree can be analyzed whole.

As far as pkgsrc is concerned, I tend to agree that it would be easier
to deal with starting different package builds in parallel (since we
maintain the tree and all of the dependancy information already - just
need to express it to make) than it would be to get all of the
packages' individual build systems to comply with this.

        - Nathan