Re: Setting up bulkbuild

[John Klos <john%klos.com@localhost>]

Hi,

> I'm interested in building stuff for VAX systems.

Great!

>   It's now in the "Scanning..." phase and manages to "scan" about 250
> packages per day, with a total of nearly 20k. So it'll be scanning for
> about three months, give or take. Are there ways to speed this up? By
> doing it parallel on several hosts, or create the scan result (I gues
> it's inter-package dependencies?) on a different (faster) host and
> then let the VAXen do actual build work (possibly proxying the
> compilation work through distcc)?

Here's a general idea of how I've been doing VAX, m68k and other bulk 
(pbulk) builds. Note that there are lots of problems with trying to use 
pbulk as documented such as "/usr/bin/awk: no program filename" issues, 
check-shlibs.mk issues (see PKG_DEVELOPER=no below), pbulk constantly 
wanting to rescan, "don't know how to make /nonexistent", and so on, so 
some of these steps might seem superfluous, but the process will fail if 
they're skipped. One day all of these should be documented in one place, 
along with their workarounds.

I run the scan part on a fast (amd64) machine. Note that I set up the 
chroot on the fast machine the same way as on the build machines, else 
we'll get lots of "don't know how to make /nonexistent" errors and wasted 
time. I then copy pbuild, presolve, pscan and friends, plus status, to a 
new pkgsrc/packages/bulklog/$architecture/meta/, then edit status to 
reflect unique times (so the reports of one build won't be in the same 
directory as another) and the correct architecture.

I then install pkgtools/libkver, then create a chroot (/usr/sandbox) on 
each bulk build machine, then, in screen, kver -r 9.0 (or 10.0) chroot 
/usr/sandbox

On my systems I tend to use /usr/local as LOCALBASE, which is advantageous 
because we don't want to install anything at all in to /usr/pkg/ in the 
chroot so it won't interfere with our builds:

mkdir -p /usr/sandbox/usr/local/lib /usr/sandbox/usr/local/sbin
cp /usr/local/lib/libkver.so /usr/sandbox/usr/local/lib/
cp /usr/local/sbin/kver /usr/sandbox/usr/local/sbin/

Next, echo "PKG_DEFAULT_OPTIONS=-pbulk-rsync" > /etc/mk.conf (in the 
sandbox) because I don't need / want to build perl and rsync when I'm 
using an NFS server that already has them.

I edit pkgsrc/mk/pbulk/pbulk.sh to change PKG_DEVELOPER to no and add this 
right after the "(cd ${PKGSRCDIR}/pkgtools/pbulk" line:

(cd ${PKGSRCDIR}/devel/distcc && PACKAGES=${TMPDIR}/packages-pbulk 
WRKOBJDIR=${TMPDIR}/obj-pbulk ${PBULKPREFIX}/bin/bmake install)

(glad to see you've found 
https://hackaday.io/project/218-speed-up-pkgsrc-on-retrocomputers/details 
:)

Just for consistency between all of my bulk build machines, I also:

groupadd -g 70 distcc
useradd -u 70 -g 70 -d /nonexistent -s /sbin/nologin distcc
useradd -u 999 -g 100 -d /home/pbulk -s /bin/sh -m pbulk

Then I run:

sh /usr/pkgsrc/mk/pbulk/pbulk.sh -n -c /etc/mk.conf

After that's done, I create a new /etc/mk.conf:

ALLOW_VULNERABLE_PACKAGES=yes
SKIP_LICENSE_CHECK=yes
OBJMACHINE=defined
PACKAGES=/usr/pkgsrc/packages/2023Q1/vax
WRKOBJDIR=/tmp/pkgbuild
.for DISTCCDEPS in sysutils/checkperms pkgtools/digest devel/distcc 
devel/popt devel/libtool-base devel/gmake devel/m4 pkgtools/cwrappers 
pkgtools/mktools
.if ${PKGPATH} == ${DISTCCDEPS}
  IGNORE_DISTCC=yes
.endif
.endfor
.if !defined(IGNORE_DISTCC)
  PKGSRC_COMPILER=distcc gcc
  DISTCC_HOSTS=10.0.20.122:3632
.endif

WRKOBJDIR is in /tmp because local disk is faster than NFS, and 
/tmp/pkgbuild needs to be 1777 permissions. DISTCC_HOSTS is, of course, 
the machine you're going to use to run distcc to help speed things up.

I then edit /usr/pbulk/bin/bulkbuild-restart and add "restart_build=yes" 
so that script_phase_pre_build and script_phase_scan are skipped.

If I'm just going to run a pbulk on this machine, I u(n)limit, then run 
/usr/pbulk/bin/bulkbuild-restart. If I'm going to run this machine as part 
of a clustered build, there are more steps, which I'll summarize here (but 
this part isn't complete).

Since we want the chroots to think they're running a very specific version 
of NetBSD, we edit /usr/sandbox/usr/pbulk/etc/pbulk.conf and change:

pbuild=/usr/pbulk/bin/pbulk-build
to
pbuild="/usr/local/sbin/kver -r 9.0 /usr/pbulk/bin/pbulk-build"

Plus the same for presolve and pscan, too.

I give each build machine a separate local address (either IPv4 or IPv6), 
configure the main system's ListenAddresses in /etc/ssh/sshd_config to not 
use the separate local address and configure the chroot's ListenAddress to 
listen. On boot I have "chroot /usr/sandbox /etc/rc.d/sshd onestart" in 
/etc/rc.local so that the chroot's sshd is always running.

I then configure and run pbulk on a fast(er) machine with a list of 
build_clients. This process has all sorts of caveats, too, which I'll have 
to document the next time I run through them. Also note that this machine 
needs to be reachable from any build machine, just as the build machines 
obviously need to be reachable from this machine (I forget why contact 
needs to be bidirectional, but with IPv6 this is easy, even over the 
Internet).

This should be a good start, I think, and I'll update with more details 
when I have a chance.

I didn't get to move from 2023Q1 to 2023Q2 for m68k, sparc, sparc64, vax, 
sh3el, mipsel, mips64eb, earmv4, or aarch64eb because my colo provider 
went out of business, but I hope to have many set back up for 2023Q3 :)

(As an aside, if anyone knows of anyone who has space, power and a little 
bandwidth for build machines, please contact me)

Also, I have a script for going through a previous quarter's binary 
packages and keeping / copying packages that haven't been updated and 
don't have prerequisites that have been updated, plus a simple edit to 
make sure that "make bin-install" doesn't pull older binary packages from 
the NetBSD servers.

When it comes to doing a cvs update and worrying about rescanning, I 
simply don't worry about rescanning. If a popular package is updated, 
and/or if there's an egregious security issue, I just manually kver / 
chroot, then "make bin-install" for that package. Perhaps this could be 
handled better in the future.

I hope this helps!

John