Re: using pkg_comp without rebuilding pkg_install or digest

[Taylor R Campbell <campbell%mumble.net@localhost>]

   Date: Thu, 26 Jun 2008 08:56:49 +0200
   From: "Julio M. Merino Vidal" <jmmv84%gmail.com@localhost>

   On Sun, Jun 15, 2008 at 3:29 AM, Taylor R Campbell 
<campbell%mumble.net@localhost> wrote:
   >  I'm running NetBSD from the netbsd-4 tag, and I'm using the
   > pkgsrc tree dated 2008Q1.  I'd like to use pkg_comp without rebuilding
   > pkg_install or digest each time I run it, since I already have binary
   > packages at the appropriate version.  For every other package, I can
   > just set DEPENDS_TARGET=bin-install in my mk.conf to use binaries when
   > they are available, but pkg_comp explicitly runs `make' and `make
   > install' for pkg_install, and `make $BUILD_TARGET' (through the
   > `pkg_build' function) for digest.  Is there a reason why pkg_comp does
   > this, or, if not, any good way to use the binaries that I already have
   > for pkg_install and digest?

   I don't remember the real reason for building digest explicitly.
   Maybe there isn't any.

   As regards pkg_install, I think it is built manually to check whether
   it is up to date or not.  If it is not up to date, the build can fail
   misteriously later on.

   You may want to bring this issue in tech-pkg with some patch to remove
   this "feature" and see if it causes problems for anyone.

Oops -- I promptly forgot about my question shortly after this
correspondence.  If you don't care to read this long message, see the
patch attached.

What I have been doing is setting nflag=no in my pkg_comp
configuration file, adding pkg_install-20080313.tgz and
digest-20080510.tgz to INSTALL_PACKAGES, and passing the `-N' option
to pkg_comp, which has some strange sequence of effects that wind up
being what I want most of the time -- installing what I specified in
INSTALL_PACKAGES without rebuilding the `default packages' from
source.  Obviously, this is suboptimal.

So, I'd like to do this the right way, without any of the ugly
kludgery above.  I thought:  If I set BUILD_TARGET=bin-install,
perhaps pkg_comp will do the right thing and install pkg_install,
digest, &c., in the chroot from binaries if they are available.

Well, if I do this, pkg_comp uses its `pkg_build' function to build
pkgtools/digest.  This function in turn uses the check_pkg_install
function to check whether pkg_install is out of date.  Then this check
actually fails: pkg_comp concludes incorrectly that pkg_install is
up-to-date (it's actually two years old in NetBSD 4.0), a mistake that
rears its head shortly when pkgsrc tries to run pkg_info with the `-E'
option, which the 20061103 version doesn't recognize.

Somehow, this error doesn't matter, and digest gets installed from a
binary, and the build proceeds.  However, since pkg_install (the newer
version) was not installed when it should have been, the build does
fail -- as jmmv said -- in mysterious ways later on.

So I think that it would be simpler just to expunge the whole
`check_pkg_install' subroutine, and to run the `pkg_build' function on
pkgtools/pkg_install when making the chroot environment, before
running `pkg_build' on anything else.  I presume that `make
bin-install' in pkgsrc won't install a binary if it's too old.

I've attached a patch to do this in pkg_comp.  It ought to be
accompanied by a small alteration to the man page, to change

   -N   With the exception of pkgtools/libkver...

to

   -N   With the exception of pkgtools/pkg_install and
        pkgtools/libkver...

Actually, this should be here anyway, because passing -N doesn't
suppress installation of pkg_install.  Perhaps there should also be
some words about ensuring that pkg_install is up-to-date in the chroot
environment.

With the patch, I still get an error message from the old version of
pkg_info, but I think it is innocuous, because the updated version of
pkg_install gets installed immediately afterward, before anything else
is done with packages.  It would be nice to do away with that error
message altogether, but doing so would require modifying other parts
of pkgsrc which I understand too little to suggest changes to.

--- ./pkg_comp.~1~      2008-07-28 14:48:41.000000000 +0000
+++ ./pkg_comp  2008-07-28 14:49:14.000000000 +0000
@@ -527,6 +527,7 @@
     # signals to umount them.
     trap "echo \"*** Process aborted ***\" ; fsumount ; exit 1" INT QUIT
 
+    makeroot_pkg_install
     makeroot_libkver
     [ "$Nflag" = "no" ] && makeroot_digest
 
@@ -604,6 +605,16 @@
     fi
 }
 
+# makeroot_pkg_install
+#
+#   Ensure pkg_install is up-to-date.
+#
+makeroot_pkg_install()
+{
+    ( PKGSRC_COMPILER=gcc; export PKGSRC_COMPILER; \
+      pkg_build pkgtools/pkg_install )
+}
+
 # makeroot_digest
 #
 #   Ensure digest is always installed, specially because PKGSRC_COMPILER
@@ -719,7 +730,7 @@
 
     # Build them
     fsmount
-    check_pkg_install    # executes copy_vulnerabilities too
+    copy_vulnerabilities
     failed=""
     for p in $pkgs; do
         echo "PKG_COMP ==> Building and installing $p"
@@ -757,36 +768,6 @@
     fi
 }
 
-# check_pkg_install
-#
-#   Ensure that the version of pkg_install inside the sandbox is new
-#   enough to work with pkgsrc.  If not, rebuild it.
-#
-check_pkg_install()
-{
-    local script
-
-    copy_vulnerabilities
-
-    # We assume filesystems are mounted!
-
-    echo "PKG_COMP ==> Checking if pkg_install is up to date"
-    script=`mktemp $DESTDIR/pkg_comp/tmp/pkg_comp-XXXX`.sh
-    init_script $script
-    cat >> $script <<EOF
-cd /usr/pkgsrc/pkgtools/pkg_comp
-fail=\$(make show-var VARNAME=PKG_FAIL_REASON)
-if echo \$fail | grep "package tools installed on this system are out of date" 
>/dev/null; then
-    echo "PKG_COMP ==> pkg_install is out of date; rebuilding"
-    cd /usr/pkgsrc/pkgtools/pkg_install
-    make && make install && make clean
-fi
-EOF
-    chmod +x $script
-    chroot $DESTDIR /pkg_comp/tmp/`basename $script`
-    rm $script
-}
-
 # ----------------------------------------------------------------------
 # install target
 # ----------------------------------------------------------------------