Re: Using pbulk

[Thomas Klausner <wiz%gatalith.at@localhost>]

On Sat, Apr 27, 2024 at 08:01:09PM +0200, Jan-Benedict Glaw wrote:
> I'd like to start mass-building packages. It seems pbulk is the tool
> to use. However, in a first attempt to set it up (using a number of
> VAX VMs), I quite struggled to get all the pieces together.
> 
>   What I'd really like to achieve is this:
> 
>   * Easily setup a master server utilizing a number of worker nodes;
>   * setup worker nodes to build actual packages. These workers shall
>     will only build for their native arch.
> 
> This is a bit of an cloud-like approach, where I'd like to use NetBSD
> current and pkgsrc current to keep watching for bootstrap issues. So
> my question here is:  Does anybody have working scripts to setup a
> master or worker node? There are some bits'n'pieces of information
> here and there, as well as the scripts at
> https://github.com/alarixnia/bulkbuild-scripts (which have a bit of a
> different direction), but it doesn't seem to be too intuitive to setup
> new builders.

Setting up pbulk falls into the following phase:

- Get a clean environment for the main pbulk process ("master"). I use
  pkgsrc/pkgtools/mksandbox for this, but any environment where you
  don't care if your ${PREFIX} gets deleted regularly is fine.

- In that environment, build and install pkgsrc/pkgtools/pbulk into a
  non-default PREFIX, let's say /usr/pbulk.

- Delete the binary packages created in that step and set up your
  mk.conf the way you like it, with the proper PREFIX.

- Get more clean environments for the worker processes. The user
  that's running the bulk build in the "master" environment needs to
  be able to login via ssh there, so set that up.

- Edit your /usr/pbulk/etc/pbulk.conf:
  master_mode=yes
  master_ip=whatever_your_ip_is
  scan_clients='client1 client2 client3 ...' # used for the scan phase
  build_clients='client1 client2' # use for the build phase
  limited_list=/path/to/file # if you just want to build some packages, not all

  "client1" etc. should be usable as arguments for ssh, i.e. 'ssh
  client1' as the user doing the bulk build should work without
  password.

Then you can run it:

/usr/pbulk/bin/bulkbuild

Cheers,
 Thomas