On Wed, Jun 15, 2005 at 04:32:27PM +0200, Geert Hendrickx wrote:
> There has been a discussion about parallel build algorithms here a while
> ago.  Someone proposed a mechanism to split up the pkg dependency tree
> in "independent" parts, and to distribute these independent trees across
> the participating machines.  I think this could be done much easier.  

That's the task I wanted to do, yes.

[...]
> 
> This mechanism requires no advanced algorithms, is easy to implement
> (the TODO file could simply be a shell-script where the "doing" and
> "done" assignments are just appended to the end), and it allows adding
> and removing build machines from the cluster at all times (when a
> machine is taken out, it should simply mark its current "doing" package
> again as "todo").  

What I wanted to do is a bit more advanced, simply because I have some
additional goals I want to meet. Beside bulk build support, I want to
use this to keep a list of packages and their requirements up-to-date.

I wanted to use two daemons for this:
(a) clusterdep
This daemon is responsible for managing the dependency list, either
by calling a helper script for a given target package or by getting them
e.g. via a socket. The former is better for a small number of packages,
the latter more efficient for bulk builds.

The output from pkgsrc is the location of the package in the tree(s),
the name of the binary package and the dependencies. Still open for me
is how to handle the effect of options (OpenBSD flavours are nice for
this :) ).

The daemon can be queried for open jobs, it checks which packages exist
already (skip this for bulk builds), does a partial topological sort and 
returns the name of a package with satisfied dependencies. It gets informed
when this is done.

(b) clusterbuild
This daemon is responsible for querying (a) and distribute the builds
across the cluster. It can be as simple as starting the job via ssh
and fetch the package back via scp.

Joerg