Re: single server: distbb or parallel pbulk?

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2013-07-01 at 10:20 BST, John Marino wrote:

> Or have I simply not found them?  Can somebody provide a technical
> recommendation on which tool we should use for this, and which
> instructions should be followed?

I have no experience of distbb, but use pbulk a lot, and at Joyent we
have developed a system where we are able to build over 10,000
packages from scratch in 11 hours, with still plenty of opportunities
for further improvements on that.

You are right though, the experience needs a lot of work and made
simpler.  In lieu of that, here are some notes on how we achieve this,
which may prove useful for you.

First, we have a number of diffs to pbulk:

  
https://github.com/joyent/pkgsrc/compare/trunk...joyent/feature/miscfix/trunk#L31R14

The notable bits are the changes to the following scripts:

  build-client-start
  scan-client-start

which adds support for chroot scripts.  With the changes integrated,
you can add something like the following to your pbulk.conf:

  chroot_create=/scripts/mksandbox
  chroot_delete=/scripts/rmsandbox
  chroot_dir=/chroot/pkgsrc-trunk
  build_chroots=4
  scan_chroots=8

This defines a couple of scripts to create and delete chroots, a base
directory where the chroots will live, and how many chroots to create
for both build and scan phases.

I have made our mksandbox and rmsandbox scripts available here:

  https://gist.github.com/jperkin/5721037

but it should be entirely possible to integrate the normal mksandbox
script from pkgsrc, or write your own.  We have a number of
requirements which meant it was simpler to write our own than hack on
the standard script.

And that's basically it.  As you can see from our pbulk diffs we also
support running concurrent instances of pbulk with the PBULK_CONF
stuff, and we use that to build any number of branches simultaneously,
all using the same pbulk base, just with different config files.

The whole system is driven by Jenkins, and is automatically triggered
by either pushes to our github tree, or by Jenkins' "cron" system
which is used to do the nightly upstream-trunk builds that you see
sent to pkgsrc-bulk every day.

I fully agree that this stuff should be simpler, and in time I will
try to both integrate some of these bits as well as document them
better, but it is possible to get very fast performance with basically
unlimited scalability from pbulk with not a huge amount of work or
money.

Hopefully that's helpful,

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com