Building packages with pkg_comp 2.1

[Robert Nestor <rnestor%icloud.com@localhost>]

I was asked to share this writeup on tech-pkg for review before possible posting to the wiki.

### Using pkg_comp 2.1 to create a pkgsrc binary repository

The _pkgtools/pkg\_comp_ package was designed as a wrapper around the _pbulk_ package and greatly simplifies the process of building and maintaining a local pkgsrc binary repository.  However, while the default setup under NetBSD works for most package builds, it fails for others and introduces some obscure runtime errors in a few of the packages it builds.  This guide explains how to configure it so that it can be used to build any package found in the _pkgsrc_ distribution.

### Prerequisites

There are the prerequisites needed by _pkg\_comp_:

* A directory containing the tarball distribution files for a system installation.  This can be found on the NetBSD installtion CD or image file used to install NetBSD on your system; for amd64 this will be the amd64/binary directory.
* A _pkgsrc_ source tree which may have been downloaded and installed when you set up your system.
* Optionally the _src_ and _xsrc_ source trees.  Only some rare packages will need this though and the setup suggested here will only use them if they have been installed.
* Optionally a tool like _misc/screen_ or _misc/tmux_ as the full build process can take a **very** long time.
* Optiopnally the _pkg\_comp4cron_ package can be installed to automate the _pkg\_comp_ build in a background _cron_ job.  (Installation may cause some confusion in setting uyp _pkg\_comp_ as described here and should probbably be deferred until after _pkg\_comp_ has been successfully installed and tested.)

Avoid automatic update of _pkgsrc_ (cron or otherwise) if you're in the middle of a build as it could lead to unfortunate results.  The _pkg\_comp_ has an option for doing an automatic update of _pkgsrc_ when run using the UPDATE\_SOURCES option. The default is to update the source tree if UPDATE\_SOURCES is not provided or set to "true".  To override this is must be set to "false", either in the /usr/pkg/etc/pkg_comp/default.conf file or speificed on the invocation command line.

### Prepare a chroot

_pkg\_comp_ runs a _pbulk_ build inside a sandbox, not on your running environment, so there's no risk of modifying any currently installed packages on your system.

The process of setting up _pkg\_comp_ is quite simple:

	# pkg_in install pkg_comp
	
> Note: _pkg\_comp_ uses _sandboxctl_ to create, configure, and destroy the underlying sandbox it uses. In doing so it provides it's own overrides to sandboxctl which are modified in the following steps.

#### Adjusting the configuration

To use _pkg\_comp_ for periodic builds, you'll need to do some minimal edits to the default configuration files.  The files should be found directly under /usr/pkg/etc/pkg_comp/, which is normally _pkg\_comp_'s "home" under NetBSD.  (If _pkg\_comp_ was built using a previous installation of _pkg\_comp_ without the changes shown below, then the files may exist under /etc/pkg_comp instead. While this works for running _pkg\_comp_ it produces packages and may not run as expected when installed on your NetBSD system.)
>Note: Changes to the **default.conf** file require that the underlying sandbox and bootstraps be reconstructed before the changes are effective.  This is forced by doing an "rm -fr packages/bootstrap* log pbulk" command in the designated pkgsrc directory before invoking _pkg\_comp_.  Changes to either the **sandbox.conf** or **extra.mk.conf** files will take effect on the next invocation of _pkg\_comp_.

\* /usr/pkg/etc/pkg_comp/sandbox.conf: This is the configuration file used with _sandboxctl_ and contains some of the overrides needed by _pkg\_comp_ in the sandbox that is created. On NetBSD the following example works and can be used once the **[path to where the NetBSD binary distribution files are located]** is correctly filled in:

	#
    # This configuration file provides a setup to build a NetBSD sandbox
    #
    SANDBOX_TYPE=netbsd-native
    SANDBOX_ROOT=/var/tmp/sandbox
    
    # Parameters for a netbsd-native sandbox
    NETBSD_NATIVE_RELEASEDIR=[path to where the NetBSD binary distribution files are located]
     
    # Parameters for a netbsd-release sandbox (not used)
    NETBSD_RELEASE_RELEASEDIR=
    NETBSD_RELEASE_SETS=
    
    # Required for building packages that use X libraries or need to build against system
    #  and/or X sources. If these directories are not present on your host system because
    #  you didn't install them, then they won't be available for use in pkg_comp and
    #  won't cause any harm here.
    pkg_comp_post_mount_hook() {
        #
        # Required to be able to build any packages that uses X or X libraries.
        #
        if [ -d /usr/X11R7 ]; then
            mkdir -p ${SANDBOX_ROOT}/usr/X11R7
            sandbox_bindfs -o ro /usr/X11R7 ${SANDBOX_ROOT}/usr/X11R7
        fi
        #
        # Optional, but needed for any packages which need access to system or X sources
        #
        if [ -d /usr/src ]; then
            mkdir -p ${SANDBOX_ROOT}/usr/src
            sandbox_bindfs -o ro /usr/src ${SANDBOX_ROOT}/usr/src
        fi
        if [ -d /usr/xsrc ]; then
            mkdir -p ${SANDBOX_ROOT}/usr/xsrc
            sandbox_bindfs -o ro /usr/xsrc ${SANDBOX_ROOT}/usr/xsrc
        fi
        exit 0
    }

\* /usr/pkg/etc/pkg_comp/default.conf: This is the configuration file used to build packages from the _pkgsrc_ tree.  On NetBSD the following will build from _pkgsrc_ and place the binary packages in the /usr/pkgsrc/packages/All directory.

    # Configuration file for pkg_comp(8); see pkg_comp.conf(5) for details.
    #
    # This configuration file provides a sample setup to build a collection of
    # packages for the local machine.
    
	# Optionally set the default for UPDATE_SOURCES to false which can be overriden when
	#  invoking a build with pkg_comp. The default is true if not specified.
	#UPDATE_SOURCES=false

    # Additional configuration files to support this setup.
    EXTRA_MKCONF="/etc/pkg_comp/extra.mk.conf"
    SANDBOX_CONFFILE="/etc/pkg_comp/sandbox.conf"
	
    # Remote VCS configuration.
    FETCH_VCS=cvs
    #CVS_ROOT=:ext:anoncvs%anoncvs.NetBSD.org@localhost:/cvsroot
    #CVS_TAG=pkgsrc
    #GIT_URL=https://github.com/NetBSD/pkgsrc.git
    #GIT_BRANCH=trunk
    
    # Host file layout.  These directories point to trees managed outside of
    # the sandbox and are exposed within the sandbox via null mounts.
    PKGSRCDIR="/usr/pkgsrc"
    DISTDIR="/usr/pkgsrc/distfiles"
    PACKAGES="${PKGSRCDIR}/packages"
    PBULK_LOG="${PACKAGES}/log"
    PBULK_PACKAGES="${PACKAGES}/pbulk"
    
    # Target file layout.  These are the directories that the built binary
    # packages will use when installed.
    LOCALBASE=/usr/pkg
    PKG_DBDIR=/var/db/pkg
    SYSCONFDIR=/usr/pkg/etc
    VARBASE=/var
    
    # Enable interactive pkgsrc development via sandbox-shell.
    PKG_DEVELOPER=yes
    
    # List of packages to build during automatic execution.
    #AUTO_PACKAGES="bash emacs kyua pkg_comp tmux"
    
    # Alternatively, use a separate list of packages to be built.  Make sure to create this
    #  list if this form is used.  The file can be in the format used for a pbulk build.
    AUTO_PACKAGES="$(grep -v '^#' '/usr/pkg/etc/pkg_comp/pbulk.list')"
	
\* /usr/pkg/etc/pkg_comp/extra.mk.conf: This is the override configuration file for mk.conf used inside the sandbox for the _pbulk_ build.

    # Configuration file to extend the mk.conf(5) created by pkg_comp(8).
    #
    # This file is included by the mk.conf file created within the sandbox by
    # pkg_comp(8).  Use this to customize the build of your packages, but do NOT
    # configure the file layout here (such as by setting LOCALBASE).
    #
    # The contents of this file are automatically guarded by BSD_PKG_MK so you don't
    # need to worry about this here.
    
    #ACCEPTABLE_LICENSES=
    #PKG_DEFAULT_OPTIONS=
 
    # These changes are optional, but they allow for a much larger body of packges to be built. 
    SKIP_LICENSE_CHECK= yes
    ALLOW_VULNERABLE_LICENSES= yes
    ALLOW_VULNERABLE_PACKAGES= yes
    
    ### other possible changes
    # Fix for building Codeblocks
    #PKG_OPTIONS.wxGTK20=gtk2
    
> Note: If the _pkg\_comp_ bootstrap environments are not found, they will be generated and saved.  This will take a little time, but upon conclusion the bootstrap configuration will be saved inside your /usr/pkgsrc/packages directory and then will be availabel to be used in future invocations speeding up the process of doing subsequent package builds.

### Building your own packages by hand

Now that your are fully installed and configured, you'll build some stuff by hand to ensure the setup works.

The simplest usage form, which involves full automation, is something like this:

    # pkg_comp -c default [-o UPDATE_SOURCE=false] auto

> Note: If your /usr/pkg/etc/pkg_comp/default.conf file doesn't contain a list of packages that will be built, (AUTO_PACKAGES=) then a build of **all** packages will be done.  Depending on your system speed this can take **weeks**.

This trivially-looking command will:

1. optionally checkout or update your copy of pkgsrc (if UPDATE_SOURCE=true);
2. create the sandbox;
3. bootstrap pkgsrc and pbulk;
4. use pbulk to build the given packages; and
5. destroy the sandbox.

After a successful invocation, you'll be left with a collection of packages in the directory you set in PACKAGES, usually the default of /usr/pkgsrc/packages/All. The pkg_summary files will also be updated so the directory can be used without further changes by _pkgin_ to install or upgrade the packages on your host.

If you'd like to restrict the set of packages to build during a manually-triggered build, provide those as additional arguments to "auto" (i.e. **auto pkgin**).  This will override the contents of AUTO_PACKAGES (which was derivied from your pbulk.list file).  The packages named in the AUTO_PACKAGES list can be either \<package name\> or \<package group/package name\>.

But what if you wanted to invoke all stages separately, bypassing the **auto**? The command above would be equivalent to:

    # pkg_comp -c default -o UPDATE_SOURCE=true fetch
    # pkg_comp -c default sandbox-create
    # pkg_comp -c default bootstrap
    # pkg_comp -c default build <package names here>
    # pkg_comp -c default sandbox-destroy

Go ahead and play with these.  You can also use the **sandbox-shell** command to interactively enter the sandbox.  See pkg_comp(8) for more details.

### Installing the resulting packages

Now that you have build your packages, you may want to install them.  Assuming your are going to use pkgtools/pkgin (and why not?), configure your local repository:

    # echo 'file:///usr/pkgsrc/packages/All' >>/usr/pkg/etc/pkgin/repositories.conf
>Note: Make sure you don't have more than one repository source active in your configuration file.  While having multiple sources specified "works", it may produce some undesired or unexpected results.

If you have a functionally working version of _pkgin_ installed on your system then all it takes to upgrade using your new set of locally built packages is:

    # pkgin update
    # pkgin upgrade

### Hints, suggestions and anomolies

* By default, _pkg\_comp_ builds packages to install with configuration files in /etc instead of /usr/pkg/etc.  This may work for many packages, but some are very sensitive to where they may look for their configuration files resulting in obscure execution problems.  This is overridden in the *default.conf* file above.

* Some packages, like net/samba, will fail to build upon exceeding an open file limit.  The easiest solution is to do a "ulimit -n 1536" before invoking the _pkg\_comp_ build.

* It's easy to set up multiple sandboxes for tracking and building different pkgsrc archives.  Assuming you'd like to track and build changes in both -current (as shown above) and 2018Q4, simply create a pkgsrc source archive for 2018Q4, such as /usr/pkgsrc-2018Q4.  Then duplicate the sandbox.conf file naming it sb2018Q4.conf and duplicate the default.conf file to 2018Q4.conf.  Now change the name of the sandbox path in sb2018Q4.conf from /var/tmp/sandbox to /var/tmp/sb2018Q4.  Then change the 2018Q4.conf file so that SANDBOX\_CONFFILE points to /usr/pkg/etc/sb2018Q4.conf and change PKGSRCDIR to point to /usr/pkgtsrc-2018Q4.  You can then invoke a 2018Q4 build with the command "pkg_comp -c 2018Q4 auto". Both environments will share the same **distfiles** source archive so multiple copies of the same source distribution won't be downloaded to your system.
> Note: Multiple sandboxes for doing builds can be run concurrently, but speed and success will depend on processor speed, size of SWAP and size of /tmp.