Re: pbulk Builds Made Easy (removal of old bulk build support)

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2014-07-14 at 23:32 BST, Aleksej Saushev wrote:

> Aleksej Saushev <asau%inbox.ru@localhost> writes:
> 
> > I was working on updating The pkgsrc Guide to describe new bulk builds,
> > and it has occurred to me that the explaining it in human language
> > isn't what a user would want actually. While exploring how to explain it
> > I have written a script that automates pbulk deployment and setup 
> > (attached).
> > Currently the whole setup of a dedicated machine (real one, VM,
> > chroot environment, FreeBSD jail, Solaris zone, or other similar thing)
> > is reduced to three commands:
> >
> > (cd /usr && cvs -Q -z3 -d anoncvs%anoncvs.netbsd.org@localhost:/cvsroot get 
> > -P pkgsrc)
> > sh pbulk.sh -n # for NetBSD, no "-n" for other systems, I used FreeBSD to 
> > test it
> > /usr/pbulk/bin/bulkbuild
> >
> > It supports setting pbulk up for building selected list of packages,
> > the routine changes to
> >
> > sh pbulk.sh -nl
> > cp pbulk.list /usr/pbulk/etc/pbulk.list
> > /usr/pbulk/bin/bulkbuild
> >
> > Thus, the script goes well along "five knobs" approach to human-machine
> > interfaces. (In addition it can be used in more complex setups,
> > I'm experimenting with automation of parallelized bulk builds
> > using the same script.)
> >
> > Now it is polished to a state that the guide would just follow the code.
> > I find explaining what the code does step by step stupid, it is only 200 
> > lines.
> > Hence I propose the following:
> >  - find a place in pkgsrc tree where we could put it;
> >  - change bulk build instructions in the Guide to suggest using the script
> >    (mentioning potential modification of it for advanced users);
> >  - schedule removal of old bulk build code.
> 
> I'm reminding about this script. You can find new version in
> http://ftp.netbsd.org/pub/NetBSD/misc/asau/pbulk-serial-7.tar.gz
> 
> Since the first publication it has been heavily tested on NetBSD,
> FreeBSD, and openSUSE. Since I haven't received any feedback,
> I'm going to put it (newer version) into mk/bulk and amend instructions
> in the Guide.
> 
> If you want to know the rationale behind the choice, here it is:
> 
> 1. The script doesn't belong to documentation. It is ready to run and
> is used in production. (E.g. all FreeBSD reports on pkgsrc-bulk are
> produced with one or another version of it.)
> 
> 2. The script doesn't belong to any package. It is meant to be run _before_
> even pkgsrc tools are bootstrapped.
> 
> 3. mk/scripts contains significantly different stuff, those scripts are
> not meant for user consumption.
> 
> Thus the closest places for it are mk/bulk and bootstrap.

I think this is a good start, a couple of comments:

 - I think we should put scripts related to pbulk in mk/pbulk, it
   already exists and is clearly separate from the legacy mk/bulk, as
   well as showing these are dedicated to pbulk and not any other bulk
   build approaches.

 - As you acknowledge in your README, your approach does not cover all
   cases.  For example it does not support chroots, at least not
   natively.  As such I'd like to see it imported as pbulk-single.sh
   or similar.

 - It would be nice if the script supported multiple runs without
   being destructive, e.g. preserve the bootstraps somewhere and
   re-use them rather than re-bootstrapping all of pbulk each time.

I will try to finish up my distributed scripts this quarter and get my
pbulk chroot diffs into a state where they can be imported, then we
should be able to offer similar scripts for easily setting up
distributed and chrooted builds.  We can then offer users a couple of
different options depending on their needs.

Thanks,

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com