Subject: install/22514: allow syspkgs to be created during unprivileged build
To: None <gnats-bugs@gnats.netbsd.org>
From: None <apb@cequrux.com>
List: netbsd-bugs
Date: 08/17/2003 16:35:07
>Number: 22514
>Category: install
>Synopsis: allow syspkgs to be created during unprivileged build
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: install-manager
>State: open
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Sun Aug 17 14:37:00 UTC 2003
>Closed-Date:
>Last-Modified:
>Originator: Alan Barrett
>Release: NetBSD 1.6W
>Organization:
Not much
>Environment:
NetBSD 1.6W/i386
>Description:
It's currently difficult to create syspkgs if you are an unprivileged
user, or if you build to DESTDIR != /. The appended patches make it
easier.
I have successfully used these patches to do complete builds
as an unprivileged user, using a command like this:
$ build.sh -D /my/destdir -T /my/tooldir -R /my/releasedir \
-u -U release syspkgs
and then installed a subset of the resulting syspkgs in a clean target
file system using commands like this:
$ su
# newfs /dev/rwd2a
# mount /dev/wd2a /my/target
# PKG_DBDIR=/my/target/var/db/syspkg pkg_add -p /my/target \
/my/releasedir/i386/binary/syspkgs/base-dhcpd-bin-*.tgz
The pkgdb.byfile.db in the target directory ends up containing entries
for "/my/target/foo/bar" instead of just "/foo/bar", but that's a
shortcoming of pkg_add which is not addressed by this PR. Everything
else, including automatic adding of dependent packages, works well
enough.
>How-To-Repeat:
Wait nearly two years hoping for a reasonable way of creating syspkgs
from an unprivileged build. Get tired of waiting.
>Fix:
Apply the following patches.
Note that regpkg and regpkgset are almost completely rewritten,
so the previous copyright notices might no longer apply.
distrib/sets/makeplist:
* Add a new "-I realprefix" arg. When making syspkgs, we need to use
"-P ${DESTDIR} -I /". ("-I" chosen for similarity to pkg_create.)
XXX: makeplist seems to be quite broken, in that it looks at the files
and directories that are actually present in DESTDIR, whereas I
thought its job was to provide an list of what *should* be there,
regardless of what is *actually* there. Fixing this seems to require
a change to the format of the "lists" files, so that they can
unambiguously identify directories.
distrib/sets/regpkg:
* Add the ability to create binary packages (*.tgz files). The new
"-t binpkgdir" option requests this action.
* Make it pay attention to DESTDIR.
* Make it work for unprivileged builds using METALOG.
* Add "force" and "update" modes.
* Add "quiet" mode. There was already a "verbose" mode.
* Add several new command line args in support of the above.
* Be much more careful about quoting.
* Make much more use of variables to refer to tools.
* Make much more use of shell functions.
* Implement a new way of choosing version numbers, controlled
by the SYSPKG_VERSION_METHOD environment variable.
SYSPKG_VERSION_METHOD=osvers: The version number is obtained
from the OS version number and the tiny version number
in the "versions" file. This gives results like like
"base-sysutil-bin-1.6W.0".
SYSPKG_VERSION_METHOD=date: The version number is obtained from
the the most recent RCSID date or the most recent mtime date of
any file in the syspkg. This ends up giving syspkg names like
This gives results like like "base-sysutil-bin-20030813".
SYSPKG_VERSION_METHOD=both: Both the above, giving results like
"base-sysutil-bin-1.6W.0.20030813".
The default is "both". For backward compatibility,
setting SYSPKG_DATES has the same effect as setting
SYSPKG_VERSION_METHOD=date.
* Add @blddep lines to the PLIST (in addition to the @pkgdep lines
that were previously added).
* Use host tools such as pax, cksum, and db, to do more or less the
same work that was previously done by pkg_add (which is not a host
tool). We don't do exactly the same work, so somebody with better
knowledge of the pkg system should review this.
distrib/sets/regpkgset:
* Sort the pkgs into dependency order before invoking regpkg.
* Accept (and pass thhrough) all the new args that regpkg takes.
distrib/sets/Makefile:
* Add new "syspkgs" target, and several helper targets for it to
use internally.
* Use "${TOOL_MTREE} -C ..." to sanitise the METALOG file before
invoking anything that will want to parse the METALOG (checkflist,
regpkgset, maketars, or installsets). Previously, maketars used to
do this itself, but its better to do it only once.
* Add a few more host tools to SETSENV.
distrib/sets/maketars:
* Remove the code that used "${MTREE} -C ..." to sanitise the plist.
We can now assume that it's already been sanitised.
distrib/sets/getdirs.awk:
* Instead of printing "./foo/bar optional" for implicitly required
directories, print "./foo/bar optional type=dir".
distrib/sets/checkflist:
* Add a check for consistency between METALOG and DESTDIR.
* Use egrep instead of awk to ignore differences that are expected.
This should be easier to maintain, and should also make it easier
for users to add their own custom exceptions by editing the
regexp.
etc/Makefile:
* After running ${TOOL_MTREE} to create directories from the
specification in .../mtree/NetBSD.dist, run ${TOOL_MTREE} again with
different args to register the new directories in the metalog.
Previously, these directories were not mentioned at all
in the metalog.
Makefile:
* Add a new "syspkgs" target, which simply chains to the "syspkgs" target
in distrib/sets/Makefile.
build.sh:
* Add a new "syspkgs" target, which simply chains to the "syspkgs" target
in the top level Makefile.
doc/BUILDING.mdoc:
* Document the new "syspkgs" target in build.sh and in the top level
Makefile.
usr.sbin/pkg_install/lib/str.c:
* Fixed a bug in dependency handling that affected syspkgs. If the
dependency does not specify a file format suffix like .t[bg]z, then
all file formats should be considered as matching.
Index: distrib/sets/makeplist
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/makeplist,v
retrieving revision 1.9
diff -u -r1.9 makeplist
--- distrib/sets/makeplist 4 Aug 2003 05:53:20 -0000 1.9
+++ distrib/sets/makeplist 17 Aug 2003 10:42:46 -0000
@@ -1,7 +1,14 @@
#!/bin/sh
#
# Print out the files in some or all lists.
-# Usage: makeplist [-a arch] [-m machine] [-s setsdir] [-p prefix] setname pkgname
+# Usage: makeplist [options] setname pkgname
+# options:
+# -a arch set arch (e.g, m68k, mips, powerpc)
+# -m machine set machine (e.g, amiga, i386, macppc)
+# -s setsdir directory to find sets
+# -p prefix prefix for package creation
+# -I realprefix prefix for eventual installation
+# setname pkgname set and package to build plist for
#
# set defaults
@@ -29,15 +36,18 @@
. ./sets.subr
setd=`dirname $0`
prefix=/
+realprefix=''
usage() {
exec 1>&2
-echo "Usage: $0 [-a arch] [-m machine] [-s setsdir] [-p prefix] setname pkgname"
+echo "Usage: $0 [options] setname pkgname"
+echo "options:"
echo " -a arch set arch (e.g, m68k, mips, powerpc) [$machine_arch]"
echo " -m machine set machine (e.g, amiga, i386, macppc) [$machine]"
echo " -s setsdir directory to find sets [$setd]"
-echo " -p prefix prefix for created plist [$prefix]"
+echo " -p prefix prefix for package creation [$prefix]"
+echo " -I realprefix prefix for eventual installation [$realprefix]"
echo " setname pkgname set and package to build plist for"
exit 1
@@ -49,6 +59,9 @@
-a*)
machine_arch=$2; shift
;;
+ -I*)
+ realprefix=$2; shift
+ ;;
-m*)
machine=$2; shift
;;
@@ -77,6 +90,9 @@
else
usage
fi
+if [ "$realprefix" = "" ]; then
+ realprefix="$prefix"
+fi
# Determine lib type.
if [ "$object_fmt" = "ELF" ]; then
@@ -106,6 +122,19 @@
SELECTDIRS="-maxdepth 0 -type d"
SELECTNONDIRS="-maxdepth 0 ! -type d"
+#
+# XXX: The "lists" do not differentiate between directories and files.
+# But we need to differentiate between them, so we do so by checking
+# what's actually present in the file system. Files or directories that
+# are listed in the "lists" but that do not exist in the file system end
+# up not appearing in our output, and this subverts a large part of the
+# purpose of the "lists".
+#
+# XXX: Given that we have to figure out what is or is not a directory
+# without assistance from the "lists", it would be much more efficient
+# to consult the metalog instead of the file system.
+#
+
cd $prefix
#
# match the directories
@@ -127,7 +156,7 @@
cd -
-echo "@cwd $prefix"
+echo "@cwd $realprefix"
if [ -s $ffilename ]; then
cat $ffilename
fi
Index: distrib/sets/regpkg
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/regpkg,v
retrieving revision 1.4
diff -u -r1.4 regpkg
--- distrib/sets/regpkg 23 Jun 2003 09:25:34 -0000 1.4
+++ distrib/sets/regpkg 17 Aug 2003 13:19:16 -0000
@@ -33,173 +33,844 @@
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
-# Usage: regpkg set pkgname
+set -e # exit on error
-SYSPKGROOT=${PKG_DBDIR:-/var/db/pkg}
-case "${SYSPKG_DBDIR}" in
-"") ;;
-*) SYSPKGROOT=${SYSPKG_DBDIR} ;;
-esac
-
-PLIST=/tmp/.PLIST.$$
-
-verbose=""
-while [ $# -gt 2 ]; do
- case $1 in
- -v) verbose=$1 ;;
- *) break ;;
- esac
- shift
-done
-
-if [ $# -ne 2 ]; then
- echo "Usage: regpkg set pkgname"
+# Usage: regpkg [options] set pkgname
+#
+# Registers a syspkg in the database directory,
+# and optionally creates a binary package.
+#
+# Options:
+# -q Quiet.
+# -v Verbose.
+# -f Force.
+# -u Update.
+# -d destdir Sets DESTDIR.
+# -t binpkgdir Create a binary package (in *.tgz format) in the
+# specified directory. Without this option, a binary
+# package is not created.
+# -M metalog Use the specified metalog file to override file
+# or directory attributes when creating a binary package.
+# -N etcdir Use the specified directory for passwd and group files.
+#
+# When -f is set: If the desired syspkg already exists, it is overwritten.
+# When -u is set: If the desired syspkg already exists, it might be
+# overwritten or left alone, depending on whether it's older
+# or newer than the files that belong to the syspkg.
+# When neither -u nor -f are set: It's an error for the desired syspkg
+# to already exist.
+
+# set defaults
+: ${AWK=awk}
+: ${CKSUM=cksum}
+: ${DB=db}
+: ${EGREP=egrep}
+: ${FGREP=fgrep}
+: ${GREP=grep}
+: ${GZIP=gzip}
+: ${IDENT=ident}
+: ${LS=ls}
+: ${MAKE=make}
+: ${MKTEMP=mktemp}
+: ${MTREE=mtree}
+: ${PAX=pax}
+: ${PKG_CREATE=pkg_create}
+: ${PRINTF=printf}
+: ${SED=sed}
+: ${SH=sh}
+: ${SORT=sort}
+: ${WC=wc}
+
+# A literal newline
+nl='
+'
+# A literal tab
+tab=' '
+
+#
+# All temporary files will go in $SCRATCH, which will be deleted on
+# exit.
+#
+SCRATCH="$( ${MKTEMP} -d "${TMPDIR:-/var/tmp}/${0##*/}.XXXXXX" )"
+if [ $? -ne 0 -o \! -d "${SCRATCH}" ]; then
+ echo >&2 "Could not create scratch directory."
exit 1
fi
-pkgset=$1
-
-pkg=$2
+#
+# cleanup() always deletes the SCRATCH directory, and might also
+# delete other files or directories.
+#
+cleanup_must_delete_binpkgfile=false
+cleanup_must_delete_dbsubdir=false
+cleanup ()
+{
+ trap '' 1 2 3
+ if ${cleanup_must_delete_binpkgfile:-false} && [ -e "${binpkgfile}" ]
+ then
+ echo >&2 "$0: deleting partially-created ${binpkgfile}"
+ rm -f "${binpkgfile}"
+ fi
+ if ${cleanup_must_delete_dbsubdir:-false} \
+ && [ -e "${SYSPKG_DB_SUBDIR}" ]
+ then
+ echo >&2 "$0: deleting partially-created ${SYSPKG_DB_SUBDIR}"
+ rm -rf "${SYSPKG_DB_SUBDIR}"
+ fi
+ rm -rf "${SCRATCH}"
+}
+trap 'cleanup' 0 1 2 3
+
+#
+# Parse command line args.
+#
+verbose=false
+quiet=false
+force=false
+update=false
+DESTDIR="${DESTDIR}"
+binpkgdir=""
+metalog=""
+etcdir=""
+SYSPKG_DB_TOPDIR=""
+pkgset=""
+pkg=""
+parse_args ()
+{
+ while [ $# -gt 2 ]; do
+ case "$1" in
+ -q) quiet=true ; verbose=false ;;
+ -v) verbose=true ; quiet=false ;;
+ -f) force=true ;;
+ -u) update=true ;;
+ -d) DESTDIR="$2" ; shift ;;
+ -d*) DESTDIR="${1#-?}" ;;
+ -t) binpkgdir="$2" ; shift ;;
+ -t*) binpkgdir="${1#-?}" ;;
+ -M) metalog="$2" ; shift ;;
+ -M*) metalog="${1#-?}" ;;
+ -N) etcdir="$2" ; shift ;;
+ -N*) etcdir="${1#-?}" ;;
+ *) break ;;
+ esac
+ shift
+ done
+ DESTDIR="${DESTDIR%/}" # delete trailing "/" if any
+ if [ \! -n "${etcdir}" ]; then
+ etcdir="${DESTDIR}/etc"
+ fi
+ if [ -n "${binpkgdir}" -a \! -d "${binpkgdir}" ]; then
+ echo >&2 "$0: ERROR: binary pkg directory ${binpkgdir} does not exist"
+ exit 1
+ fi
+ if $force; then
+ ERRWARN="WARNING"
+ else
+ ERRWARN="ERROR"
+ fi
+ if $update; then
+ ERRWARNNOTE="NOTE"
+ else
+ ERRWARNNOTE="${ERRWARN}"
+ fi
+ #
+ # SYSPKG_DB_TOPDIR is the top level directory for registering
+ # syspkgs. It defaults to ${DESTDIR}/var/db/syspkg, but can be
+ # overridden by environment variables SYSPKG_DBDIR or PKG_DBDIR.
+ #
+ # Note that this corresponds to the default value of PKG_DBDIR
+ # set in .../distrib/syspkg/mk/bsd.syspkg.mk.
+ #
+ SYSPKG_DB_TOPDIR=${SYSPKG_DBDIR:-${PKG_DBDIR:-${DESTDIR}/var/db/syspkg}}
+
+ if [ $# -ne 2 ]; then
+ echo "Usage: regpkg [options] set pkgname"
+ exit 1
+ fi
-case $verbose in
--v) echo "Making PLIST for \"$pkgset\" set and \"$pkg\" package" ;;
-esac
+ pkgset="$1"
+ pkg="$2"
+}
+
+#
+# make_PLIST() creates a skeleton PLIST from the pkgset description.
+#
+# The result is stored in the file ${PLIST}.
+#
+PLIST="${SCRATCH}/PLIST"
+make_PLIST ()
+{
+ if $verbose ; then
+ echo "Making PLIST for \"$pkg\" package (part of ${pkgset} set)"
+ fi
+ prefix="${DESTDIR:-/}"
+ realprefix=/
+ ${SH} ./makeplist -p "${prefix}" -I "${realprefix}" \
+ "${pkgset}" "${pkg}" \
+ >"$PLIST" 2>"${SCRATCH}/makeplist-errors"
+ if [ -s "${SCRATCH}/makeplist-errors" ]; then
+ # XXX "find" invoked from makeplist sometimes reports
+ # errors about missing files or directories, and
+ # makeplist ignores the errors. Catch them here.
+ echo >&2 "$0: ERROR: makeplist reported errors for ${pkg}:"
+ cat >&2 "${SCRATCH}/makeplist-errors"
+ exit 1 # this is fatal even with $force.
+ fi
+}
-# create the skeleton PLIST from the pkgset description
-./makeplist $pkgset $pkg > $PLIST
+#
+# init_allfiles() converts the PLIST (which contains relative filenames)
+# into a list of absolute filenames. Directories are excluded from the
+# result.
+#
+# The result is stored in the variable ${allfiles}.
+#
+allfiles=''
+init_allfiles ()
+{
+ [ -f "${PLIST}" ] || make_PLIST
+ allfiles="$( ${AWK} '
+ BEGIN { destdir = "'"${DESTDIR%/}"'" }
+ /^@cwd/ { prefix = $2; next }
+ /^@dirrm/ { next }
+ { printf("%s%s%s\n", destdir, prefix, $0) }' "$PLIST" )"
+}
-# create the pkg tiny version
-case "${SYSPKG_DATES}" in
-"") tinyvers=`awk '$1 ~ '/$pkg/' { print $2 }' versions`
- case "$tinyvers" in
- "") tinyvers=0
- ;;
- esac
+#
+# init_newestfile() finds the newest file (most recent mtime).
+#
+# The result is stored in the variable ${newestfile}.
+#
+newestfile=''
+init_newestfile ()
+{
+ [ -s "${allfiles}" ] || init_allfiles
+ newestfile="$( ${LS} -1dt $allfiles | ${SED} '1q' )"
+}
+
+#
+# Various ways of getting parts of the syspkg version number:
+#
+# get_osvers() - get the OS version number
+# get_tinyvers() - get the tiny version number from the "versions" file
+# get_newest_rcsid_date() - get the newest RCS date
+# get_newest_mtime_date() - get the newest file modification date
+# get_newest_date() - get date from rcsid or mtime
+#
+get_osvers ()
+{
if [ -f ../../sys/conf/osrelease.sh ]; then
- osvers=`sh ../../sys/conf/osrelease.sh`
+ osvers="$( ${SH} ../../sys/conf/osrelease.sh )"
method=osreleases
else
osvers=`uname -r`
method=uname
fi
- t=$osvers.$tinyvers
- ;;
-*) args=`awk '
- /^@cwd/ { prefix = $2; next }
- /^@dirrm/ { next }
- { printf("%s%s\n", prefix, $0) }' $PLIST`
- # first try for any RCS identifiers in the files
- t=0
- case "$args" in
+ echo "$osvers"
+}
+get_tinyvers ()
+{
+ tinyvers=`${AWK} '$1 ~ '/$pkg/' { print $2 }' versions`
+ case "$tinyvers" in
+ "") tinyvers=0
+ ;;
+ esac
+ echo "${tinyvers}"
+}
+get_newest_rcsid_date()
+{
+ [ -s "${allfiles}" ] || init_allfiles
+
+ # Old RCS identifiers might have 2-digit years, so we match
+ # both YY/MM/DD and YYYY/MM/DD.
+ newest=0
+ case "${allfiles}" in
"") ;;
- *) t=`ident $args 2>/dev/null | awk '
+ *) newest="$( ${IDENT} ${allfiles} 2>/dev/null | ${AWK} '
BEGIN { last = 0 }
$2 == "crt0.c,v" { next }
- NF == 8 { t = $4; gsub("/", "", t); if (t > last) last = t; }
- END { print last }'`
+ NF == 8 && \
+ $4 ~ /^([0-9][0-9])?[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9]$/ \
+ { t = $4; gsub("/", "", t);
+ if (t > last) last = t; }
+ END { print last }' )"
method=ident
;;
esac
- case "$t" in
- 0) # we need the last mtime of the files which make up the package
- t=`env TZ=UTC LOCALE=C ls -lT $args | awk '
- BEGIN { newest = 0 }
- {
- t = $9 "";
- if ($6 == "Jan") t = t "01";
- if ($6 == "Feb") t = t "02";
- if ($6 == "Mar") t = t "03";
- if ($6 == "Apr") t = t "04";
- if ($6 == "May") t = t "05";
- if ($6 == "Jun") t = t "06";
- if ($6 == "Jul") t = t "07";
- if ($6 == "Aug") t = t "08";
- if ($6 == "Sep") t = t "09";
- if ($6 == "Oct") t = t "10";
- if ($6 == "Nov") t = t "11";
- if ($6 == "Dec") t = t "12";
- if ($7 < 10) t = t "0";
- t = t $7;
- #these next two lines add the 24h clock onto the date
- #gsub(":", "", $8);
- #t = sprintf("%s.%4.4s", t, $8);
- if (t > newest) newest = t;
- }
- END { print newest }'`
+ echo "${newest}"
+}
+get_newest_mtime_date ()
+{
+ [ -s "${newestfile}" ] || init_newestfile
+
+ # There should be exactly one line in in the input to awk,
+ # so we could simplify the awk program.
+ newest="$( env TZ=UTC LOCALE=C ${LS} -lT "${newestfile}" | ${AWK} '
+ BEGIN { newest = 0 }
+ {
+ t = $9 "";
+ if ($6 == "Jan") t = t "01";
+ if ($6 == "Feb") t = t "02";
+ if ($6 == "Mar") t = t "03";
+ if ($6 == "Apr") t = t "04";
+ if ($6 == "May") t = t "05";
+ if ($6 == "Jun") t = t "06";
+ if ($6 == "Jul") t = t "07";
+ if ($6 == "Aug") t = t "08";
+ if ($6 == "Sep") t = t "09";
+ if ($6 == "Oct") t = t "10";
+ if ($6 == "Nov") t = t "11";
+ if ($6 == "Dec") t = t "12";
+ if ($7 < 10) t = t "0";
+ t = t $7;
+ #these next two lines add the 24h clock onto the date
+ #gsub(":", "", $8);
+ #t = sprintf("%s.%4.4s", t, $8);
+ if (t > newest) newest = t;
+ }
+ END { print newest }' )"
+ echo "${newest}"
+}
+get_newest_date ()
+{
+ newest="$( get_newest_rcsid_date )"
+ case "${newest}" in
+ ""|0) newest="$( get_newest_mtime_date )"
method=ls
;;
+ *) method=rcsid
+ ;;
esac
- ;;
-esac
+ echo "${newest}"
+}
-# print version number that we're using
-case "$verbose" in
--v) echo "$pkg - $t version using $method method" ;;
-esac
-
-# create the directory and minimal contents
-SYSPKGDIR=${SYSPKGROOT}/$pkg-$t
-if [ -d ${SYSPKGDIR} ]; then
- echo "There is already a $pkg-$t package installed (${SYSPKGDIR})"
- exit 1
-fi
+#
+# choose_version_number() chooses the syspkg version number using
+# whatever method SYSPKG_VERSION_METHOD asks for.
+#
+# The result is stored in the variables ${t} and ${method}.
+#
+method=''
+t=''
+choose_version_number ()
+{
+ # backward compatibility: if SYSPKG_DATES is set,
+ # and SYSPKG_VERSION_METHOD is not set, then
+ # behave as if SYSPKG_VERSION_METHOD=date.
+ case "${SYSPKG_DATES+set}:${SYSPKG_VERSION_METHOD-unset}" in
+ set:unset) SYSPKG_VERSION_METHOD=date ;;
+ esac
-mkdir -p ${SYSPKGDIR}
+ case "${SYSPKG_VERSION_METHOD:-both}" in
+ osvers) t="$( get_osvers ).$( get_tinyvers )"
+ ;;
+ date) t="$( get_newest_date )"
+ ;;
+ both)
+ t1="$( get_osvers ).$( get_tinyvers )"
+ m1="${method}"
+ t2="$( get_newest_date )"
+ m2="${method}"
+ t="${t1}.${t2}"
+ method="${m1}.${m2}"
+ ;;
+ *) echo >&2 "$0: ERROR: bogus value for SYSPKG_VERSION_METHOD"
+ exit 1
+ ;;
+ esac
-# add the dependencies
-awk '$1 ~ '/$pkg/' { print $2 }' deps | sort | \
- awk '{ print "@pkgdep " $1 "-[0-9]*" }' >> $PLIST
-
-# create the comment
-comment=`awk '$1 ~ '/$pkg/' { print substr($0, length($1) + 2) }' comments`
-case "$comment" in
-"") echo "***WARNING ***: no comment for \"$pkg\"" 2>&1
- comment="System package for $pkg"
- ;;
-esac
-echo "$comment" > ${SYSPKGDIR}/+COMMENT
-
-# create the description
-descr=`awk '$1 ~ '/$pkg/' { print substr($0, length($1) + 2) }' descrs`
-case "$descr" in
-"") echo "***WARNING ***: no description for \"$pkg\"" 2>&1
- descr="System package for $pkg"
- ;;
-esac
-echo "$descr" > ${SYSPKGDIR}/+DESC
-printf "\nHomepage:\nhttp://www.NetBSD.org/\n" >> ${SYSPKGDIR}/+DESC
-
-# create the build information
-{
-echo "OPSYS=`uname -s`"
-echo "OS_VERSION=`uname -r`"
-make -f- all <<EOF
+ # print version number that we're using
+ if $verbose; then
+ echo "$pkg - $t version using $method method"
+ fi
+}
+
+#
+# register_syspkg() registers the syspkg in ${SYSPKG_DB_TOPDIR}.
+# This involves creating the subdirectory ${SYSPKG_DB_SUBDIR}
+# and populating it with several files.
+#
+register_syspkg ()
+{
+ cleanup_must_delete_dbsubdir=true
+ [ -n "${SYSPKG_DB_SUBDIR}" ] || exit 2 # XXX should never happen
+ mkdir -p "${SYSPKG_DB_SUBDIR}"
+
+ #
+ # Add the dependencies.
+ #
+ # We always add a "@pkgdep" line for each prerequisite package.
+ #
+ # If the prerequisite pkg is already registered (as it should be
+ # if our caller is doing things in the right order), then we put
+ # its exact version number in a "@blddep" line.
+ #
+ # XXX: We could store something better than a -[0-9]* wildcard
+ # in the "deps" file (probably in an extra field), or guess a
+ # tighter wildcard based on the OS version.
+ #
+ ${AWK} '$1 ~ '/$pkg/' { print $2 }' deps | ${SORT} | \
+ while read dep ; do
+ pkgdep="${dep}-[0-9]*"
+ echo "@pkgdep ${pkgdep}"
+ blddep="$( cd "${SYSPKG_DB_TOPDIR}" && echo $pkgdep )"
+ case "${blddep}" in
+ *\*) # wildcard did not match anything
+ echo >&2 "$0: WARNING: ${pkg} depends on '${pkgdep}' but there is no matching syspkg in ${SYSPKG_DB_TOPDIR}"
+ ;;
+ *\ *) # wildcard matched more than once.
+ echo >&2 "$0: $ERRWARN: ${pkg} depends on '${pkgdep}' but there are multiple matching syspkgs in ${SYSPKG_DB_TOPDIR}"
+ $force || exit 1
+ # XXX: Assume that the last match is the most recent.
+ # This might be wrong, because of differences between
+ # lexical sorting and numeric sorting.
+ lastmatch="${blddep##* }"
+ echo "@blddep ${lastmatch}"
+ ;;
+ *) # exactly one match
+ echo "@blddep ${blddep}"
+ ;;
+ esac
+ done >>"${PLIST}"
+
+ # create the comment
+ comment="$( ${AWK} '$1 ~ '/$pkg/' \
+ { print substr($0, length($1) + 2) }' \
+ comments )"
+ case "$comment" in
+ "") echo >&2 "$0: WARNING: no comment for \"$pkg\""
+ comment="System package for $pkg"
+ ;;
+ esac
+ echo "$comment" > "${SYSPKG_DB_SUBDIR}/+COMMENT"
+
+ # create the description
+ descr="$( ${AWK} '$1 ~ '/$pkg/' { print substr($0, length($1) + 2) }' \
+ descrs )"
+ case "$descr" in
+ "") echo >&2 "$0: WARNING: no description for \"$pkg\"" 2>&1
+ descr="$comment"
+ ;;
+ esac
+ echo "$descr" > "${SYSPKG_DB_SUBDIR}/+DESC"
+ ${PRINTF} "\nHomepage:\nhttp://www.NetBSD.org/\n" >> "${SYSPKG_DB_SUBDIR}/+DESC"
+
+ # create the build information
+ {
+ # XXX: Are these variables supposed to describe the host or the
+ # target? They currently describe the host.
+ echo "OPSYS=`uname -s`"
+ echo "OS_VERSION=`uname -r`"
+ ${MAKE} -f- all <<EOF
.include <bsd.own.mk>
all:
@echo OBJECT_FMT=${OBJECT_FMT}
@echo MACHINE_ARCH=${MACHINE_ARCH}
@echo MACHINE_GNU_ARCH=${MACHINE_GNU_ARCH}
EOF
-echo "_PKGTOOLS_VER=`pkg_create -V`"
-} > ${SYSPKGDIR}/+BUILD_INFO
+ echo "_PKGTOOLS_VER=`pkg_create -V`"
+ } > "${SYSPKG_DB_SUBDIR}/+BUILD_INFO"
-# test for attributes
-args=""
-attrs=`awk '$1 ~ '/$pkg/' { print substr($0, length($1) + 2) }' attrs`
-for a in "$attrs"; do
- case "$attrs" in
- "") ;;
- preserve) echo "$pkg-$t" > ${SYSPKGDIR}/+PRESERVE
- args="$args -n ${SYSPKGDIR}/+PRESERVE"
- ;;
- esac
-done
+ # test for attributes
+ args=""
+ attrs=`${AWK} '$1 ~ '/$pkg/' { print substr($0, length($1) + 2) }' attrs`
+ for a in "$attrs"; do
+ case "$attrs" in
+ "") ;;
+ preserve) echo "$pkg-$t" > "${SYSPKG_DB_SUBDIR}/+PRESERVE"
+ args="$args -n ${SYSPKG_DB_SUBDIR}/+PRESERVE"
+ ;;
+ esac
+ done
+
+ #
+ # Finish registering the syspkg.
+ #
+ # Below are two implementations of more or less the same thing:
+ # one using pkg_create. and one not using pkg_create. We
+ # actually use the implementation that does not use pkg_create,
+ # because pkg_create is not a host tool, so we cannot expect it
+ # to be present during cross builds.
+ #
+ # XXX: Perhaps we should detect whether a usable pkg_create is
+ # available, and if so then choose the implementation that uses
+ # pkg_create.
+ #
+ # The invocation of pkg_create in
+ # finish_register_using_pkg_create essentially does two things:
+ #
+ # * Creates ${SYSPKGSIR}/+CONTENTS from ${PLIST}, by adding an
+ # "@name" line and a lot of "@comment MD5:" lines. We could
+ # do this using "cksum", which is already a host tool.
+ #
+ # * Updates ${SYSPKG_DB_TOPDIR}/pkgdb.byfile.db. We could
+ # skip this for cross builds, or we could do it using "db",
+ # which is already a host tool.
+ #
+ # finish_register_by_hand does both those jobs without using
+ # pkg_create. create_contents_file_by_hand does the first part,
+ # and update_pkgdb_by_hand does the second part.
+ #
+ finish_register_using_pkg_create ()
+ {
+ # pkg_create is not a host tool, so we should not do
+ # this, unless we know that it's not a cross build.
+ PKG_DBDIR="${SYSPKG_DB_TOPDIR}" \
+ ${PKG_CREATE} -v -c "${SYSPKG_DB_SUBDIR}/+COMMENT" \
+ -d "${SYSPKG_DB_SUBDIR}/+DESC" \
+ $args \
+ -p "${prefix}" -I "${realprefix}" \
+ -f "$PLIST" -l -b /dev/null \
+ -B "${SYSPKG_DB_SUBDIR}/+BUILD_INFO" \
+ -s /dev/null -S /dev/null -O "$pkg-$t" \
+ > "${SYSPKG_DB_SUBDIR}/+CONTENTS"
+ }
+ create_contents_file_by_hand ()
+ {
+ # Create the +CONTENTS file.
+ {
+ rcsid='$NetBSD$'
+ date="$( TZ=UTC date '+%Y-%m-%d %H:%M' )"
+ user="${USER:-root}"
+ host="$( hostname )"
+ echo "@name ${pkg}-${t}"
+ echo "@comment Packaged at ${date} by ${user}@${host} using ${rcsid}"
+ # Move the @pkgdep and @blddep lines up, so that
+ # they are easy to see when people do "less
+ # ${DESTDIR}/var/db/syspkg/*/+CONTENTS".
+ ${EGREP} '^(@pkgdep|@blddep)' "${PLIST}" || true
+ while read line ; do
+ case "${line}" in
+ @pkgdep*|@blddep*)
+ # already handled by grep
+ ;;
+ @*) echo "${line}"
+ ;;
+ *) echo "${line}"
+ file="${DESTDIR}${line}"
+ if [ -f "${file}" -a -r "${file}" ];
+ then
+ md5sum="$( ${CKSUM} -n -m "${file}" \
+ | ${AWK} '{print $1}'
+ )"
+ echo "@comment MD5:${md5sum}"
+ fi
+ ;;
+ esac
+ done <"${PLIST}"
+ } >"${SYSPKG_DB_SUBDIR}/+CONTENTS"
+ }
+ update_pkgdb_by_hand ()
+ {
+ dbfile="${SYSPKG_DB_TOPDIR}/pkgdb.byfile.db"
+ dbtype="btree"
+ db_opts=''
+ # XXX: cross builds should really add "-E B" or "-E L"
+ # to db_opts, but we don't do that yet.
+ if $update || $force ; then
+ # overwriting an existing entry is not an error
+ db_opts="${db_opts} -R"
+ fi
+ if ! $verbose; then
+ # don't print all the keys added to the database
+ db_opts="${db_opts} -q"
+ fi
+
+ # Transform $PLIST into a form to be used as keys in
+ # ${dbfile}. The results look like absolute paths,
+ # but they are really relative to ${DESTDIR}.
+ #
+ # "@dirrm ." -> "/"
+ # "@dirrm foo/bar" -> "/foo/bar"
+ # "@dirrm ./foo/bar" -> "/foo/bar"
+ # "foo/bar/baz" -> "/foo/bar/baz"
+ # "./foo/bar/baz" -> "/foo/bar/baz"
+ #
+ dblist="${SCRATCH}/dblist"
+ ${AWK} '/^@dirrm \.\// {gsub("^.", "", $2); print $2; next}
+ /^@dirrm \.$/ {print "/"; next}
+ /^@dirrm/ {print "/" $2; next}
+ /^@/ {next}
+ /^\.\// {gsub("^.", "", $0); print $0; next}
+ /./ {print "/" $0; next}' \
+ <"${PLIST}" >"${dblist}"
+ # Add all the path names to the database.
+ ${AWK} '{print $1 "\t" "'"${pkg}-${t}"'"}' <"${dblist}" \
+ | ${DB} -w ${db_opts} -F "${tab}" -f - "${dbtype}" "${dbfile}"
+ }
+ finish_register_by_hand ()
+ {
+ create_contents_file_by_hand
+ update_pkgdb_by_hand
+ }
+ # Here we choose which implementation of finish_register_* to use.
+ finish_register_by_hand
+
+ $quiet || echo "Registered ${pkg}-${t} in ${SYSPKG_DB_TOPDIR}"
+
+ cleanup_must_delete_dbsubdir=false
+}
+
+#
+# create_syspkg_tgz() creates the *.tgz file for the package.
+#
+# The output file is ${binpkgdir}/${pkg}-${t}.tgz.
+#
+create_syspkg_tgz ()
+{
+ #
+ # pkg_create does not understand metalog files, so we have to
+ # use pax directly.
+ #
+ # We create two specfiles: specfile_overhead describes the
+ # special files that are part of the package system's metadata
+ # (+CONTENTS, +COMMENT, +DESCR, and more); and specfile_payload
+ # describes the files and directories that we actually want as
+ # part of the package's payload.
+ #
+ # We then use the specfiles to create a compressed tarball that
+ # contains both the overhead files and the payload files.
+ #
+ # There's no trivial way to get a single pax run to do
+ # everything we want, so we run pax twice, with a different
+ # working directory and a different specfile each time.
+ #
+ # We could conceivably make clever use of pax's "-s" option to
+ # get what we want from a single pax run with a single (more
+ # complicated) specfile, but the extra trouble doesn't seem
+ # warranted.
+ #
+ cleanup_must_delete_binpkgfile=true
+ specfile_overhead="${SCRATCH}/spec_overhead"
+ specfile_payload="${SCRATCH}/spec_payload"
+ tarball_uncompressed="${SCRATCH}/tarball_uncompressed"
+
+ # Create a specfile for all the overhead files (+CONTENTS and
+ # friends).
+ {
+ plusnames_first="${SCRATCH}/plusnames_first"
+ plusnames_rest="${SCRATCH}/plusnames_rest"
+
+ # Ensure that the first few files are in the same order
+ # that "pkg_create" would have used, just in case anything
+ # depends on that. Other files in alphabetical order.
+ SHOULD_BE_FIRST="+CONTENTS +COMMENT +DESC"
+ (
+ cd "${SYSPKG_DB_SUBDIR}"
+ for file in ${SHOULD_BE_FIRST}; do
+ [ -e "./$file" ] && echo "$file"
+ done >"${plusnames_first}"
+ ${LS} -1 | ${FGREP} -v -f "${plusnames_first}" \
+ >"${plusnames_rest}" \
+ || true
+ )
+
+ # Convert the file list to specfile format, and override the
+ # uid/gid/mode.
+ {
+ echo ". optional type=dir"
+ ${AWK} '{print "./" $0 " type=file uid=0 gid=0 mode=0444"
+ }' "${plusnames_first}" "${plusnames_rest}"
+ } >"${specfile_overhead}"
+ }
+
+ # Create a specfile for the payload of the package.
+ {
+ spec1="${SCRATCH}/spec1"
+ spec2="${SCRATCH}/spec2"
+
+ # Transform $PLIST into simple specfile format:
+ #
+ # "@dirrm ." -> "."
+ # "@dirrm foo/bar" -> "./foo/bar"
+ # "@dirrm ./foo/bar" -> "./foo/bar"
+ # "foo/bar/baz" -> "./foo/bar/baz"
+ # "./foo/bar/baz" -> "./foo/bar/baz"
+ #
+ # Ignores @cwd lines. This should be safe, given how
+ # makeplist works.
+ ${AWK} '/^@dirrm \.\// {print $2; next}
+ /^@dirrm \.$/ {print "."; next}
+ /^@dirrm/ {print "./" $2; next}
+ /^@/ {next}
+ /^\.\// {print $0; next}
+ /./ {print "./" $0; next}' \
+ <"${PLIST}" >"${spec1}"
+
+ # If metalog was specified, attributes from metalog override
+ # attributes in the file system.
+ {
+ if [ -n "${metalog}" ]; then
+ ${AWK} -f join.awk "${spec1}" "${metalog}"
+ else
+ cat "${spec1}"
+ fi
+ } >"${spec2}"
+
+ #
+ # If a file or directory to was mentioned explicitly
+ # in $PLIST but not mentioned in $metalog, then the
+ # file or directory will not be mentioned in $spec2.
+ # This is an error, and means that the metalog was
+ # not built correctly.
+ #
+ if [ -n "${metalog}" ]; then
+ # If spec1 and spec2 both have the same number of lines,
+ # assume that all is well. If not, report on the
+ # differences.
+ n1=$( ${WC} -l "${spec1}" | ${AWK} '{print $1}' )
+ n2=$( ${WC} -l "${spec2}" | ${AWK} '{print $1}' )
+ if [ x"${n1}" != x"${n2}" ]; then
+ names2="${SCRATCH}/names2"
+ ${AWK} '{print $1}' <"${spec2}" >"${names2}"
+ cat >&2 <<EOM
+$0: $ERRWARN: The metalog file (${metalog}) does not
+ contain entries for the following files or directories
+ which should be part of the ${pkg} syspkg:
+EOM
+ ${FGREP} -v -f "${names2}" "${spec1}" >&2
+ $force || exit 1
+ fi
+ fi
+
+ # Add lines (tagged "optional") for any implicit directories.
+ #
+ # For example, if we have a file ./foo/bar/baz, then we add
+ # "./foo/bar optional type=dir", "./foo optional type=dir",
+ # and ". optional type=dir", unless those directories were
+ # already mentioned explicitly.
+ #
+ ${AWK} -f getdirs.awk "${spec2}" \
+ | ${SORT} -u >"${specfile_payload}"
+ }
+
+ # Use two pax invocations followed by gzip to create
+ # the tgz file.
+ #
+ # Remove any leading "./" from path names, because that
+ # could confuse tools that work with binary packages.
+ (
+ cd "${SYSPKG_DB_SUBDIR}" && \
+ ${PAX} -O -w -d -N${etcdir} -M '-s,^\./,,' \
+ -f "${tarball_uncompressed}" \
+ <"${specfile_overhead}"
+ )
+ (
+ cd "${DESTDIR:-/}" && \
+ ${PAX} -O -w -d -N${etcdir} -M '-s,^\./,,' \
+ -a -f "${tarball_uncompressed}" \
+ <"${specfile_payload}"
+ )
+ ${GZIP} -9 <"${tarball_uncompressed}" >"${binpkgfile}"
+
+ $quiet || echo "Created ${binpkgfile}"
+ cleanup_must_delete_binpkgfile=false
+
+}
+
+#
+# do_register_syspkg() registers the syspkg if appropriate.
+#
+# If SYSPKG_DB_SUBDIR already exists, that might be an error, depending
+# on $force and $update flags.
+#
+do_register_syspkg ()
+{
+ [ -n "${SYSPKG_DB_SUBDIR}" ] || exit 2 # XXX should never happen
+
+ delete_and_reregister ()
+ {
+ echo >&2 "$0: $ERRWARNNOTE: deleting and re-registering $pkg-$t"
+ cleanup_must_delete_dbsubdir=true
+ rm -rf "${SYSPKG_DB_SUBDIR}"
+ register_syspkg
+ }
+
+ if [ -d "${SYSPKG_DB_SUBDIR}" ]; then
+ echo >&2 "$0: $ERRWARNNOTE: $pkg-$t is already registered in ${SYSPKG_DB_TOPDIR}"
+ if $force; then
+ delete_and_reregister
+ elif $update; then
+ #
+ # If all files in SYSPKG_DB_SUBDIR are newer
+ # than all files in the pkg, then do nothing.
+ # Else delete and re-register the pkg.
+ #
+ [ -n "${newestfile}" ] || init_newestfile
+ if [ -n "${newestfile}" ]; then
+ case "$( find "${SYSPKG_DB_SUBDIR}" -type f \
+ ! -newer "${newestfile}" -print )" \
+ in
+ "") ;;
+ *) delete_and_reregister ;;
+ esac
+
+ else
+ # No files in the pkg? (This could happen
+ # if a pkg contains only directories.)
+ # Do nothing.
+ fi
+ else
+ exit 1
+ fi
+ else
+ register_syspkg
+ fi
+}
-pkg_create -v -c ${SYSPKGDIR}/+COMMENT \
- -d ${SYSPKGDIR}/+DESC \
- $args \
- -f $PLIST -l -b /dev/null -B ${SYSPKGDIR}/+BUILD_INFO \
- -s /dev/null -S /dev/null -O $pkg-$t \
- > ${SYSPKGDIR}/+CONTENTS
+#
+# do_create_syspkg_tgz() creates the the binary pkg (*.tgz) if
+# appropriate.
+#
+# If binpkgfile already exists, that might be an error, depending on
+# $force and $update flags.
+#
+do_create_syspkg_tgz ()
+{
+ [ -n "${binpkgfile}" ] || exit 2 # XXX should never happen
+
+ delete_and_recreate ()
+ {
+ echo >&2 "$0: $ERRWARNNOTE: deleting and re-creating $pkg-$t.tgz"
+ rm -f "${binpkgfile}"
+ create_syspkg_tgz
+ }
+
+ if [ -e "${binpkgfile}" ]; then
+ echo >&2 "$0: $ERRWARNNOTE: $pkg-$t.tgz already exists in ${binpkgdir}"
+ if $force; then
+ delete_and_recreate
+ elif $update; then
+ #
+ # If all files in SYSPKG_DB_SUBDIR are older
+ # than ${binpkgfile}, then do nothing.
+ # Else delete and re-create the tgz.
+ #
+ case "$( find "${SYSPKG_DB_SUBDIR}" -type f \
+ -newer "${binpkgfile}" -print )" \
+ in
+ "") ;;
+ *) delete_and_recreate ;;
+ esac
+ else
+ exit 1
+ fi
+ else
+ create_syspkg_tgz
+ fi
+}
+
+####################
+# begin main program
+
+parse_args ${1+"$@"}
+make_PLIST
+choose_version_number
+SYSPKG_DB_SUBDIR="${SYSPKG_DB_TOPDIR}/$pkg-$t"
+do_register_syspkg
+if [ -n "${binpkgdir}" ]; then
+ binpkgfile="${binpkgdir}/${pkg}-${t}.tgz"
+ do_create_syspkg_tgz
+fi
-rm $PLIST
+exit 0
Index: distrib/sets/regpkgset
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/regpkgset,v
retrieving revision 1.1
diff -u -r1.1 regpkgset
--- distrib/sets/regpkgset 12 Jun 2003 20:04:00 -0000 1.1
+++ distrib/sets/regpkgset 17 Aug 2003 11:52:09 -0000
@@ -33,20 +33,62 @@
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
-# Usage: regpkgset set
+set -e # exit on error
-verbose=""
+# Usage: regpkgset [options] set
+#
+# Options:
+# -q Quiet.
+# -v Verbose.
+# -f Force.
+# -u Update.
+# -d destdir Sets DESTDIR.
+# -t binpkgdir Create a binary package (in *.tgz format) in the
+# specified directory. Without this option, a binary
+# package is not created.
+# -M metalog Use the specified metalog file to override file
+# or directory attributes when creating a binary package.
+# -N etcdir Use the specified directory for passwd and group files.
+# set defaults
+: ${AWK=awk}
+: ${TSORT=tsort}
+
+# A literal newline
+nl='
+'
+
+verbose=false
+quiet=false
+force=false
+update=false
+pkgdir=""
+metalog=""
+etcdir=""
+all_options=""
while [ $# -gt 1 ]; do
case $1 in
- -v) verbose=$1 ;;
+ -q) quiet=true ; verbose=false ;;
+ -v) verbose=true ; quiet=false ;;
+ -f) force=true ;;
+ -u) update=true ;;
+ -d) DESTDIR="$2" ; all_options="${all_options} $1" ; shift ;;
+ -d*) DESTDIR="${1#-?}" ;;
+ -t) pkgdir="$2" ; all_options="${all_options} $1" ; shift ;;
+ -t*) pkgdir="${1#-?}" ;;
+ -M) metalog="$2" ; all_options="${all_options} $1" ; shift ;;
+ -M*) metalog="${1#-?}" ;;
+ -N) etcdir="$2" ; all_options="${all_options} $1" ; shift ;;
+ -N*) etcdir="${1#-?}" ;;
*) break ;;
esac
+ all_options="${all_options} $1"
shift
done
+export DESTDIR
if [ $# -lt 1 ]; then
- echo "Usage: regpkgset pkgset..."
+ echo "Usage: regpkgset [options] set ..."
exit 1
fi
@@ -55,10 +97,49 @@
*) list=$@ ;;
esac
-for pkgset in $list; do
- for pkg in `./listpkgs $pkgset`; do
- ./regpkg $verbose $pkgset $pkg
- done
+#
+# For each pkgset mentioned in $list, get a list of all pkgs in the pkgset.
+#
+# Sort all the pkgs into dependency order (with prerequisite pkgs before
+# pkgs that depend on them).
+#
+# Invoke ./regpkg for each pkg, taking care to do it in dependency
+# order. If there were any pkgs for which we failed to find dependency
+# information, handle them at the end.
+#
+pkgs="$( for pkgset in $list; do
+ ./listpkgs $pkgset
+ done )"
+tsort_input="$( ${AWK} '{print $2 " " $1}' <./deps )"
+tsort_output="$( echo "${tsort_input}" | ${TSORT} )"
+for pkg in ${tsort_output} ; do
+ case "${nl}${pkgs}${nl}" in
+ *"${nl}${pkg}${nl}"*)
+ # We want this pkg.
+ pkgset="${pkg%%-*}"
+ $verbose && echo "$0: registering $pkg"
+ ./regpkg ${all_options} $pkgset $pkg || exit 1
+ ;;
+ *) # pkg is mentioned in ${tsort_output} but not in ${pkgs}.
+ # We do not want this pkg.
+ ;;
+ esac
+done
+for pkg in ${pkgs} ; do
+ case "${nl}${tsort_output}${nl}" in
+ *"${nl}${pkg}${nl}"*)
+ # pkg was in the tsort output, so it would have been
+ # handled above.
+ ;;
+ *) # This pkg was not in the tsort output.
+ # This is probably an error, but process the
+ # pkg anyway.
+ echo >&2 "$0: WARNING: ${pkg} is not mentioned in deps file"
+ pkgset="${pkg%%-*}"
+ $verbose && echo "$0: registering $pkg"
+ ./regpkg ${all_options} $pkgset $pkg || exit 1
+ ;;
+ esac
done
exit 0
Index: distrib/sets/Makefile
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/Makefile,v
retrieving revision 1.29
diff -u -r1.29 Makefile
--- distrib/sets/Makefile 4 Aug 2003 05:53:20 -0000 1.29
+++ distrib/sets/Makefile 16 Aug 2003 20:55:44 -0000
@@ -10,8 +10,13 @@
SETSENV= DESTDIR=${DESTDIR:Q} \
MACHINE=${MACHINE:Q} MACHINE_ARCH=${MACHINE_ARCH:Q} \
- CKSUM=${TOOL_CKSUM:Q} MAKE=${MAKE:Q} MTREE=${TOOL_MTREE:Q} \
- PAX=${TOOL_PAX:Q}
+ CKSUM=${TOOL_CKSUM:Q} \
+ DB=${TOOL_DB:Q} \
+ MAKE=${MAKE:Q} \
+ MKTEMP=${TOOL_MKTEMP:Q} \
+ MTREE=${TOOL_MTREE:Q} \
+ PAX=${TOOL_PAX:Q} \
+ TSORT=${TSORT:Q}
print_machine: .PHONY
@echo ${MACHINE}
@@ -48,26 +53,41 @@
.if ${MKUNPRIVED} == "no"
METALOG.unpriv=
+METALOG.sanitised.unpriv=
.else
METALOG.unpriv= -M ${METALOG}
+METALOG.sanitised= ${METALOG}.sanitised
+METALOG.sanitised.unpriv= -M ${METALOG.sanitised}
+.endif
+
+.if ${MKUNPRIVED} == "no"
+sanitise_METALOG:
+.else
+sanitise_METALOG: ${METALOG.sanitised}
+.PRECIOUS: ${METALOG.sanitised}
+${METALOG.sanitised}: ${METALOG}
+ ${TOOL_MTREE} -C -k all -N ${NETBSDSRCDIR}/etc <${METALOG} >${.TARGET}
.endif
.PRECIOUS: checkflist
-checkflist: check_DESTDIR
- ${SETSENV} ${HOST_SH} ${.CURDIR}/checkflist ${METALOG.unpriv}
+checkflist: check_DESTDIR sanitise_METALOG
+ ${SETSENV} ${HOST_SH} ${.CURDIR}/checkflist ${METALOG.sanitised.unpriv}
.PRECIOUS: checkflist-x11
checkflist-x11: check_DESTDIR
${SETSENV} ${HOST_SH} ${.CURDIR}/checkflist -x11
-.PRECIOUS: maketars
-maketars: check_DESTDIR check_RELEASEDIR
.if defined(DESTDIR) && ${DESTDIR} != ""
- ${MAKE} checkflist${CHECKFLISTFLAGS}
+checkflist_if_DESTDIR: checkflist${CHECKFLISTFLAGS}
+.else
+checkflist_if_DESTDIR:
.endif
+
+.PRECIOUS: maketars
+maketars: check_DESTDIR check_RELEASEDIR sanitise_METALOG checkflist_if_DESTDIR
mkdir -p ${RELEASEDIR}/${MACHINE}/binary/sets
${SETSENV} ${HOST_SH} ${.CURDIR}/maketars -d ${DESTDIR:S,^$,/,} \
- ${METALOG.unpriv} -N ${NETBSDSRCDIR}/etc \
+ ${METALOG.sanitised.unpriv} -N ${NETBSDSRCDIR}/etc \
-t ${RELEASEDIR}/${MACHINE}/binary/sets ${MAKETARSETS}
.PRECIOUS: makesrctars
@@ -81,18 +101,39 @@
${SETSENV} ${MAKESUMS} -t ${RELEASEDIR}/${MACHINE}/binary/sets
.PRECIOUS: installsets
-installsets: check_DESTDIR
+installsets: check_DESTDIR sanitise_METALOG
.if !defined(INSTALLDIR)
@echo "setenv INSTALLDIR before doing that!"
@false
.endif
${SETSENV} ${HOST_SH} ${.CURDIR}/maketars -d ${DESTDIR:S,^$,/,} \
- ${METALOG.unpriv} -N ${NETBSDSRCDIR}/etc \
+ ${METALOG.sanitised.unpriv} -N ${NETBSDSRCDIR}/etc \
-i ${INSTALLDIR} ${INSTALLSETS}
+SYSPKGSETS?= all
+REGPKG.verbose?= # -q, -v, or empty
+REGPKG.force?= # -f, or empty
+REGPKG.update:= ${MKUPDATE:tl:Nno:C/..*/-u/}
+.PRECIOUS: makesyspkgs
+makesyspkgs: check_DESTDIR check_RELEASEDIR sanitise_METALOG \
+ checkflist_if_DESTDIR
+ mkdir -p ${RELEASEDIR}/${MACHINE}/binary/syspkgs
+ ${SETSENV} ${HOST_SH} ${.CURDIR}/regpkgset \
+ ${REGPKG.verbose} ${REGPKG.force} ${REGPKG.update} \
+ -d ${DESTDIR:S,^$,/,} ${METALOG.sanitised.unpriv} \
+ -N ${NETBSDSRCDIR}/etc \
+ -t ${RELEASEDIR}/${MACHINE}/binary/syspkgs ${SYSPKGSETS}
+
+.PRECIOUS: makesyspkgsums
+makesyspkgsums: check_RELEASEDIR
+ ${SETSENV} ${MAKESUMS} -t ${RELEASEDIR}/${MACHINE}/binary/syspkgs
+
#
# MAIN ENTRY POINTS
#
+
+syspkgs: makesyspkgs .WAIT makesyspkgsums
+ @true
sets: maketars .WAIT makesums
@true
Index: distrib/sets/maketars
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/maketars,v
retrieving revision 1.41
diff -u -r1.41 maketars
--- distrib/sets/maketars 3 Jul 2003 06:43:02 -0000 1.41
+++ distrib/sets/maketars 17 Aug 2003 10:45:14 -0000
@@ -139,20 +139,6 @@
}
trap cleanup 0 2 3 13 # EXIT INT QUIT PIPE
-if [ -n "$metalog" ]; then
- echo "Parsing $metalog"
- (
- cat ${etcdir}/mtree/NetBSD.dist
- echo "/unset all"
- cat $metalog
- ) | ${MTREE} -C -k all -N ${etcdir} > $SDIR/metalog
- rv=$?
- if [ $rv -ne 0 ]; then
- echo "${prog}: mtree failed, exiting"
- exit $rv
- fi
-fi
-
GZIP=-9 # for pax -z
export GZIP
@@ -170,7 +156,7 @@
> $SDIR/flist.$setname.full
(
echo "/set uname=root gname=wheel"
- awk -f join.awk $SDIR/flist.$setname.full $SDIR/metalog
+ awk -f join.awk $SDIR/flist.$setname.full $metalog
) > $SDIR/plist.$setname
else
mv $SDIR/flist.$setname $SDIR/plist.$setname
Index: distrib/sets/getdirs.awk
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/getdirs.awk,v
retrieving revision 1.2
diff -u -r1.2 getdirs.awk
--- distrib/sets/getdirs.awk 19 May 2002 13:32:44 -0000 1.2
+++ distrib/sets/getdirs.awk 8 Aug 2003 17:57:30 -0000
@@ -72,6 +72,6 @@
{
for (file in dirs) {
if (! (file in items))
- print file " optional"
+ print file " optional type=dir"
}
}
Index: distrib/sets/checkflist
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/distrib/sets/checkflist,v
retrieving revision 1.21
diff -u -r1.21 checkflist
--- distrib/sets/checkflist 13 Aug 2003 00:03:00 -0000 1.21
+++ distrib/sets/checkflist 17 Aug 2003 12:23:48 -0000
@@ -2,7 +2,7 @@
#
# $NetBSD: checkflist,v 1.21 2003/08/13 00:03:00 lukem Exp $
#
-# Verify output of makeflist against contents of $DESTDIR.
+# Verify output of makeflist against contents of $DESTDIR and $metalog.
if [ -z "$DESTDIR" ]; then
echo "DESTDIR must be set"
@@ -11,6 +11,20 @@
prog=${0##*/}
+: ${AWK=awk}
+: ${COMM=comm}
+: ${EGREP=egrep}
+: ${FIND=find}
+: ${SORT=sort}
+: ${MKTEMP=mktemp}
+
+# XXX Why don't we just use MKTEMP as follows, instead of the next
+# 20 lines ?
+#SDIR="$( ${MKTEMP} -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXXXXXX" )"
+#if [ $? -ne 0 -o \! -d "${SDIR}" ]; then
+# echo >&2 "Could not create scratch directory."
+# exit 1
+#fi
# Make sure we don't loop forever if mkdir will always fail.
if [ ! -d /tmp ]; then
@@ -30,7 +44,7 @@
SDIR=${SDIR_BASE}.${SDIR_SERIAL}
mkdir -m 0700 ${SDIR} && break
SDIR_SERIAL=$((${SDIR_SERIAL} + 1))
-done
+done
es=0
cleanup()
@@ -78,42 +92,76 @@
shift
done
-if [ -n "$metalog" ]; then
- case "$metalog" in
- ${DESTDIR}/*)
- # Metalog would be noticed, so make sure it gets
- # ignored.
- metalog="./${metalog#${DESTDIR}/}"
- ;;
- *)
- metalog=""
- esac
+#
+# Exceptions to flist checking (all begin with "./"):
+#
+# * ignore var/db/syspkg and its contents
+# * ignore $metalog
+# * ignore METALOG
+#
+IGNORE_REGEXP="^\./var/db/syspkg(\$|/)"
+if [ -n "${metalog}" ]; then
+ ml="${metalog#${DESTDIR}/}"
+ ml2="METALOG"
+ IGNORE_REGEXP="${IGNORE_REGEXP}|^\./${ml}\$|^\./${ml2}\$"
fi
+#
+# Make three lists:
+# * $SDIR/files: files present in DESTDIR.
+# * $SDIR/flist: files mentioned in flist;
+# * $SDIR/mlist: files mentioned in metalog;
+#
+( cd $DESTDIR && ${FIND} $origin \( -type d -o -type f -o -type l \) -print ) \
+ | ${SORT} -u | ${EGREP} -v -e "${IGNORE_REGEXP}" >"${SDIR}/files"
+sh makeflist $xargs $dargs \
+ | ${SORT} -u | ${EGREP} -v -e "${IGNORE_REGEXP}" >"${SDIR}/flist"
+if [ -n "${metalog}" ]; then
+ ${AWK} '{print $1}' <"${metalog}" \
+ | ${SORT} -u | ${EGREP} -v -e "${IGNORE_REGEXP}" >"${SDIR}/mlist"
+fi
-sh makeflist $xargs $dargs > $SDIR/flist
+#
+# compare DESTDIR with METALOG, and report on differences.
+#
+if [ -n "$metalog" ]; then
+ ${COMM} -23 $SDIR/files $SDIR/mlist > $SDIR/missing
+ ${COMM} -13 $SDIR/files $SDIR/mlist > $SDIR/extra
-(
- cd $DESTDIR
- find $origin \( -type d -o -type f -o -type l \)
-) | (
- while read line; do
- case "$line" in
- $metalog)
- ;;
- *)
- echo $line
- ;;
- esac
- done
-) | sort > $SDIR/files
+ if [ -s $SDIR/extra ]; then
+ echo ""
+ echo "======= Extra files in METALOG ========="
+ echo "Files in METALOG but missing from DESTDIR."
+ echo "File was deleted after installation ?"
+ echo "------------------------------------------"
+ cat $SDIR/extra
+ echo "========= end of extra files ==========="
+ echo ""
+ es=1
+ fi
+
+ if [ -s $SDIR/missing ]; then
+ echo ""
+ echo "======= Files missing from METALOG ====="
+ echo "Files in DESTDIR but missing from METALOG."
+ echo "File installed but not registered in METALOG ?"
+ echo "------------------------------------------"
+ cat $SDIR/missing
+ echo "======== end of missing files =========="
+ echo ""
+ es=1
+ fi
+fi
-comm -23 $SDIR/flist $SDIR/files > $SDIR/missing
-comm -13 $SDIR/flist $SDIR/files > $SDIR/extra
+#
+# compare flist with DESTDIR, and report on differences.
+#
+${COMM} -23 $SDIR/flist $SDIR/files > $SDIR/missing
+${COMM} -13 $SDIR/flist $SDIR/files > $SDIR/extra
if [ -s $SDIR/extra ]; then
echo ""
- echo "============ extra files ==============="
+ echo "======= Extra files in DESTDIR ========="
echo "Files in DESTDIR but missing from flist."
echo "File is obsolete or flist is out of date ?"
echo "------------------------------------------"
@@ -125,7 +173,7 @@
if [ -s $SDIR/missing ]; then
echo ""
- echo "=========== missing files =============="
+ echo "====== Missing files in DESTDIR ========"
echo "Files in flist but missing from DESTDIR."
echo "File wasn't installed ?"
echo "------------------------------------------"
Index: etc/Makefile
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/etc/Makefile,v
retrieving revision 1.255
diff -u -r1.255 Makefile
--- etc/Makefile 18 Jul 2003 08:26:03 -0000 1.255
+++ etc/Makefile 17 Aug 2003 12:31:45 -0000
@@ -307,10 +307,22 @@
.endif
distrib-dirs: check_DESTDIR
+# XXX: If METALOG=${DESTDIR}/METALOG (as is normal), and if ${DESTDIR}
+# does not yet exist (as often happens), then the first INSTALL_DIR
+# below will create ${DESTDIR} but will fail to register an entry for
+# "." in the metalog. We ignore the problem, because the TOOL_MTREE
+# commands below will soon register an entry for "." in the metalog.
${INSTALL_DIR} -o root -g wheel -m 755 ${BASE_PKG} ${DESTDIR}
+# XXX: It would be nice if a single mtree invocation could both
+# append to the metalog and do real work. Instead, we have to
+# repeat the command twice in slightly different ways.
${TOOL_MTREE} -def ${.CURDIR}/mtree/NetBSD.dist -N ${.CURDIR} \
-p ${DESTDIR}/ -U ${TOOL_MTREE.unpriv}
-
+.if ${MKUNPRIVED} != "no"
+ ${TOOL_MTREE} -def ${.CURDIR}/mtree/NetBSD.dist -N ${.CURDIR} \
+ -p ${DESTDIR}/ -C -k all | \
+ awk '/ optional/ {next} // {print}' | ${METALOG.add}
+.endif
# release, snapshot --
# Build a full distribution including kernels & install media.
Index: Makefile
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/Makefile,v
retrieving revision 1.221
diff -u -r1.221 Makefile
--- Makefile 26 Jul 2003 17:10:25 -0000 1.221
+++ Makefile 17 Aug 2003 10:31:41 -0000
@@ -59,6 +59,8 @@
# Populate ${RELEASEDIR}/${MACHINE}/binary/sets from ${DESTDIR}
# sourcesets:
# Populate ${RELEASEDIR}/source/sets from ${NETBSDSRCDIR}
+# syspkgs:
+# Populate ${RELEASEDIR}/${MACHINE}/binary/syspkgs from ${DESTDIR}
#
# Targets invoked by `make build,' in order:
# cleandir: cleans the tree.
@@ -273,7 +275,7 @@
# Create sets from $DESTDIR or $NETBSDSRCDIR into $RELEASEDIR
#
-.for tgt in sets sourcesets
+.for tgt in sets sourcesets syspkgs
${tgt}:
(cd ${.CURDIR}/distrib/sets && ${MAKE} $@)
.endfor
Index: build.sh
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/build.sh,v
retrieving revision 1.113
diff -u -r1.113 build.sh
--- build.sh 11 Aug 2003 19:26:04 -0000 1.113
+++ build.sh 17 Aug 2003 10:32:23 -0000
@@ -131,6 +131,7 @@
do_install=false
do_sets=false
do_sourcesets=false
+ do_syspkgs=false
do_params=false
# Create scratch directory
@@ -362,6 +363,7 @@
releasekernel=conf Install kernel built by kernel=conf to RELEASEDIR
sets Create binary sets in RELEASEDIR/MACHINE/binary/sets
sourcesets Create source sets in RELEASEDIR/source/sets
+ syspkgs Create syspkgs in RELEASEDIR/MACHINE/binary/syspkgs
params Display various make(1) parameters
Options:
@@ -572,7 +574,7 @@
usage
;;
- makewrapper|obj|tools|build|distribution|release|sets|sourcesets|params)
+ makewrapper|obj|tools|build|distribution|release|sets|sourcesets|syspkgs|params)
;;
kernel=*|releasekernel=*)
@@ -966,7 +968,7 @@
buildtools
;;
- obj|build|distribution|release|sets|sourcesets|params)
+ obj|build|distribution|release|sets|sourcesets|syspkgs|params)
${runcmd} "${makewrapper}" ${parallel} ${op} ||
bomb "Failed to make ${op}"
statusmsg "Successful make ${op}"
Index: doc/BUILDING.mdoc
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/doc/BUILDING.mdoc,v
retrieving revision 1.28
diff -u -r1.28 BUILDING.mdoc
--- doc/BUILDING.mdoc 8 Aug 2003 01:52:24 -0000 1.28
+++ doc/BUILDING.mdoc 17 Aug 2003 12:40:13 -0000
@@ -834,6 +834,17 @@
Create source sets of the source tree into
.Sy RELEASEDIR Ns Pa /source/sets .
.
+.It Sy syspkgs
+Create syspkgs from
+.Sy DESTDIR
+into
+.Sy RELEASEDIR/MACHINE Ns Pa /binary/syspkgs .
+Should be run after
+.Dq make distribution
+(as
+.Dq make build
+does not install all of the required files).
+.
.It Sy release
Do a
.Dq make distribution ,
@@ -990,6 +1001,10 @@
.It Sy sourcesets
Perform
.Dq make sourcesets .
+.
+.It Sy syspkgs
+Perform
+.Dq make syspkgs .
.
.El
.
Index: usr.sbin/pkg_install/lib/str.c
===================================================================
RCS file: /usr/netbsd/anoncvs/main/src/usr.sbin/pkg_install/lib/str.c,v
retrieving revision 1.42
diff -u -r1.42 str.c
--- usr.sbin/pkg_install/lib/str.c 5 Jan 2003 21:27:34 -0000 1.42
+++ usr.sbin/pkg_install/lib/str.c 17 Aug 2003 13:37:03 -0000
@@ -470,9 +470,11 @@
/* we need to match pattern and suffix separately, in case
* each is a different pattern class (e.g. dewey and
- * character class (.t[bg]z)) */
+ * character class (.t[bg]z)). If pat_sfx is the empty
+ * string then let it match any file_sfx.
+ */
if (pmatch(tmp_pattern, tmp_file)
- && pmatch(pat_sfx, file_sfx)) {
+ && (pat_sfx[0] == '\0' || pmatch(pat_sfx, file_sfx))) {
if (match) {
match(dp->d_name, data);
/* return value ignored for now */
(end of patches)
>Release-Note:
>Audit-Trail:
>Unformatted: