Re: FreeBSD's package cluster

[John Klos <john%ziaspace.com@localhost>]

> I would like to comment on this really quickly.  While I would be happy 
> to see glunix get fixed in the pkg collection, the honest truth is its 
> mostly useful as a reference tool, and a whole parallel system.  It is 
> *not* good for parallel compiling.  Originally thats what I wanted to do 
> with it when I ported it.  After discovering that it wasn't what I 
> needed at all, I ended up writing ClusterIt.  ClusterIt was designed 
> specifically for parallel compiling, and I think you will have more luck 
> going that route.  Specifically what you probably want is jsd/jsh, and 
> barrier/barrierd.

I'd like to point out something that hasn't been discussed much in this 
thread. FreeBSD supports systems which are generally fast. It makes sense 
to cluster a few sparc64s or ia64s or whathaveyou. However, NetBSD 
supports many archs, and clustering a gaggle of VAX is not exactly 
practical (although it'd be unmatchable for bragging rights).

What'd be neat to see is a set of tools that could have one (fast) machine 
preprocess, as much as possible, some or all of the packages, then have 
the slower, real processors do the compiling. ClusterIt seems like it'd be 
an ideal way to use one machine to coordinate all of them. It'd also be 
neat to have a way to have multiple machines of the same arch work on 
different, non-interdependent packages.

I started work on something like this a couple of years ago, but never got 
too far. I wanted to have a database keep track of all of the packages in 
the tree, all that had already been built for each arch / OS version, and 
have it, when the tree changes, track the changes and tell each slave 
machine to build whatever's new. It was going to use ssh in a chroot so 
that machines which are not local to one another could still be used.

Has anyone worked on anything like this? Would there be interest?

John Klos