pkgsrc-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[pkgsrc/trunk]: pkgsrc Rework the README.html generation code. Major changes...
details: https://anonhg.NetBSD.org/pkgsrc/rev/7b5f92e7a7e5
branches: trunk
changeset: 493975:7b5f92e7a7e5
user: dmcmahill <dmcmahill%pkgsrc.org@localhost>
date: Tue May 17 21:46:59 2005 +0000
description:
Rework the README.html generation code. Major changes are:
- completely redo the code which decides on the machine architecture,
operating system, and operating system version for the binary packages.
The old way just used to directory names to take a guess. The new
way creates a cache file containing meta-data for all the binary packages
in each "All" directory. This cache file is consulted when generating
the lists of available binary packages. The meta-data is obtained with
pkg_info so it should always be correct even if you do something silly
like mix OS_VERSION or MACHINE_ARCH packages up in the same directory.
Among the benefits are: works when PACKAGES is not $PKGSRC/packages,
works with a more or less arbitrary subdirectory structure, works
when there are subdirectories for multiple operating systems.
This portion of the fix should address PR25390.
The cache files are only updated when the contents of an "All" directory
changes or if the cache file format changes. There is some room for
improving the updating of the cache files, but its not too bad the way
it is.
- fix up some of the awk code so that generadme.awk works with Solaris
nawk as well as NetBSD's nawk and gawk (for pre-2.0 systems).
- remove some "if ! foo" shell constructs to increase portability.
- be more consistent with what variables get passed to mkreadme from
make and which ones are determined automatically. Mostly this meant
moving stuff into mkreadme to make it easier to run it standalone.
diffstat:
Makefile | 18 +--
mk/bsd.pkg.mk | 14 +-
mk/scripts/binpkg-cache | 312 +++++++++++++++++++++++++++++++++++++++++++++++
mk/scripts/genreadme.awk | 283 ++++++++++++++++++++++++++++--------------
mk/scripts/mkreadme | 121 ++++++-----------
5 files changed, 554 insertions(+), 194 deletions(-)
diffs (truncated from 1096 to 300 lines):
diff -r 7664c2722674 -r 7b5f92e7a7e5 Makefile
--- a/Makefile Tue May 17 21:34:29 2005 +0000
+++ b/Makefile Tue May 17 21:46:59 2005 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile,v 1.70 2005/05/16 19:59:54 wiz Exp $
+# $NetBSD: Makefile,v 1.71 2005/05/17 21:46:59 dmcmahill Exp $
#
# tools used by this Makefile
@@ -78,22 +78,10 @@
_README_TYPE_FLAG?= none
README.html: .PRECIOUS
.if ${_README_TYPE_FLAG} == "--ftp" || ${_README_TYPE_FLAG} == "--cdrom"
- @if [ -e ${PACKAGES} ]; then \
- cd ${PACKAGES}; \
- case `pwd` in \
- ${.CURDIR}/packages) \
- MULTIARCH=; \
- ;; \
- *) \
- MULTIARCH=--multi-arch; \
- ;; \
- esac; \
- cd ${.CURDIR} ; \
- fi; \
${SETENV} TMPDIR=${TMPDIR:U/tmp}/mkreadme \
- BMAKE=${MAKE} AWK=${AWK} EXPR=${EXPR} \
+ BMAKE=${MAKE} \
./mk/scripts/mkreadme --pkgsrc ${.CURDIR} \
- --packages ${PACKAGES} ${_README_TYPE_FLAG} $$MULTIARCH \
+ --packages ${PACKAGES} ${_README_TYPE_FLAG} \
--prune
.else
@${ECHO} "ERROR: please do not use the README.html target directly."
diff -r 7664c2722674 -r 7b5f92e7a7e5 mk/bsd.pkg.mk
--- a/mk/bsd.pkg.mk Tue May 17 21:34:29 2005 +0000
+++ b/mk/bsd.pkg.mk Tue May 17 21:46:59 2005 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: bsd.pkg.mk,v 1.1667 2005/05/17 06:31:00 jlam Exp $
+# $NetBSD: bsd.pkg.mk,v 1.1668 2005/05/17 21:46:59 dmcmahill Exp $
#
# This file is in the public domain.
#
@@ -3994,16 +3994,18 @@
${AWK} -f ../../mk/scripts/genreadme.awk \
builddependsfile=/dev/null \
dependsfile=/dev/null \
+ AWK=${AWK} \
+ CMP=${CMP} \
DISTDIR=${DISTDIR} \
- MACHINE_ARCH=${MACHINE_ARCH} \
- MULTIARCH=$$MULTIARCH \
- OPSYS=${OPSYS} \
- OS_VERSION=${OS_VERSION} \
+ GREP=${GREP} \
PACKAGES=${PACKAGES} \
+ PKG_INFO="${PKG_INFO}" \
PKG_SUFX=${PKG_SUFX} \
PKG_URL=${PKG_URL} \
- PKGREPOSITORYSUBDIR=${PKGREPOSITORYSUBDIR} \
PKGSRCDIR=${.CURDIR:C|/[^/]*/[^/]*$||} \
+ SED=${SED} \
+ SETENV=${SETENV} \
+ SORT=${SORT} \
TMPDIR=${TMPDIR:U/tmp} \
SINGLEPKG=${PKGPATH} \
$@.tmp1
diff -r 7664c2722674 -r 7b5f92e7a7e5 mk/scripts/binpkg-cache
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/mk/scripts/binpkg-cache Tue May 17 21:46:59 2005 +0000
@@ -0,0 +1,312 @@
+#!/bin/sh
+#
+# $NetBSD: binpkg-cache,v 1.1 2005/05/17 21:46:59 dmcmahill Exp $
+#
+# Script for generating a cache file with information about
+# all binary packages contained in a directory.
+#
+# Copyright (c) 2005 The NetBSD Foundation, Inc.
+# All rights reserved.
+#
+# This code is derived from software contributed to The NetBSD Foundation
+# by Dan McMahill.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# 3. All advertising materials mentioning features or use of this software
+# must display the following acknowledgement:
+# This product includes software developed by the NetBSD
+# Foundation, Inc. and its contributors.
+# 4. Neither the name of The NetBSD Foundation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+#
+
+
+TMPDIR=${TMPDIR:-/tmp}
+PACKAGES=${PACKAGES:-/usr/pkgsrc/packages/}
+AWK=${AWK:-awk}
+CMP=${CMP:-cmp}
+GREP=${GREP:-grep}
+PKG_INFO=${PKG_INFO:-pkg_info}
+PKG_SUFX=${PKG_SUFX:-.tgz}
+SED=${SED:-sed}
+SORT=${SORT:-sort}
+STAT=${STAT:-stat}
+
+cachefile=.pkgcache
+cacheversion=20050428
+
+prompt="----> "
+tab=" "
+
+tmpd=${TMPDIR}/pkg-cache.$$
+mkdir -m 0700 ${tmpd}
+if test $? -ne 0 ; then
+ echo "ERROR: Could not create temporary directory ${tmpd}"
+ echo "Either you do not have write permission to ${tmpd} or"
+ echo "${tmpd} already exists"
+ exit 1
+fi
+
+prog=$0
+
+usage(){
+ echo "$prog - Generates cache files for each directory containing binary"
+ echo " packages. This cache file can then be used by the README.html"
+ echo " generation code to avoid having to call pkg_info(1) over and over"
+ echo " on the same binary package."
+ echo " "
+ echo "Usage: $prog [-d|--debug] [-v|--verbose] [-p|--packages <dir>]"
+ echo " "
+ echo " $prog -h|--help"
+ echo " "
+ echo " $prog -V|--version"
+ echo " "
+ echo "The options supported by $prog are: "
+ echo " "
+ echo " -d|--debug Enables debugging output"
+ echo " "
+ echo " -f|--database <file> Writes the database into file specified by <file>"
+ echo " "
+ echo " -h|--help Displays this help message"
+ echo " "
+ echo " -p|--packages <dir> Specifies the top level directory to be searched"
+ echo " for binary packages."
+ echo " "
+ echo " -v|--version Displays the version of this script and exits."
+ echo " "
+ echo "Example: $prog -v --packages /usr/pkgsrc/packages"
+ echo " "
+}
+
+clean_and_exit0(){
+ rm -fr ${tmpd}
+ exit 0
+}
+
+clean_and_exit1(){
+ rm -fr ${tmpd}
+ exit 1
+}
+
+all_cache_files=""
+
+process_binpkg_dir(){
+ all_cache_files="${all_cache_files} ${d}/${cachefile}"
+
+ need_update=no
+ if test -f ${d}/${cachefile} ; then
+ stale_entries=`find ${d} -type f -name \*${PKG_SUFX} -newer ${d}/${cachefile} -print`
+
+ # FIX_ME
+ #
+ # We also should find cache entries for files which no longer exist
+ # and nuke them. Right now we simply declare the entire cache out
+ # of date. Once we implement incremental updates to the cache,
+ # we need to remove the entries but not mark the entire cache as
+ # bad.
+ if test "X${DEBUG}" = "Xyes" ; then
+ echo " Checking for cache entries with no corresponding pkg."
+ fi
+ # get the list of what pkgs belong in the cache
+ rm -f ${tmpd}/pkg_list ${tmpd}/cache_pkg_list
+ ls ${d}/*${PKG_SUFX} | ${SED} "s;^${PACKAGES}/*;;g" | ${SORT} > ${tmpd}/pkg_list
+
+ # and get the list of what is in the cache
+ ${AWK} '/pkgcache_begin/ {gsub(/pkgcache_begin[ \t]*/, ""); print}' \
+ ${d}/${cachefile} | ${SORT} > ${tmpd}/cache_pkg_list
+
+ if ${CMP} -s ${tmpd}/pkg_list ${tmpd}/cache_pkg_list ; then
+ if test "X${DEBUG}" = "Xyes" ; then
+ echo " No extra cache entries in ${d}/${cachefile}"
+ fi
+ else
+ echo " Entries found in ${d}/${cachefile} but no packages found"
+ need_update=yes
+ fi
+ else
+ stale_entries=""
+ fi
+
+ if test "X${force}" = "Xyes" -o "X${need_update}" = "Xyes" ; then
+ need_update=yes
+ echo "${tab}Forcing rebuild of cache ${d}/${cachefile}."
+ elif test ! -f ${d}/${cachefile} ; then
+ need_update=yes
+ echo "${tab}Missing cache file. ${d}/${cachefile} will be generated."
+ elif test -n "${stale_entries}" ; then
+ need_update=yes
+ echo "${tab}Stale cache file. ${d}/${cachefile} will be regenerated."
+ else
+ ${GREP} "pkgcache_version ${cacheversion}" ${d}/${cachefile} >/dev/null 2>&1
+ if test $? -ne 0 ; then
+ need_update=yes
+ echo "${tab}Old version cache file. ${d}/${cachefile} will be regenerated."
+ else
+ echo "${tab}Cache file ${d}/${cachefile} is up to date."
+ fi
+ fi
+
+ # FIX_ME
+ # We should use stale_entries in a way where we only update the
+ # cache file entries corresponding to these if we're rebuilding
+ # due to stale entries. That should save a good bit of time.
+ #
+ if test "X${need_update}" = "Xyes" ; then
+ echo "pkgcache_version ${cacheversion}" > ${tmpd}/${cachefile}
+ for f in ${d}/*${PKG_SUFX} ; do
+ fn=`echo $f | ${SED} "s;^${PACKAGES}/*;;g"`
+ if test "X${DEBUG}" = "Xyes" ; then
+ echo " Adding ${fn} (${f}) to the cache"
+ fi
+ echo " " >> ${tmpd}/${cachefile}
+ # stat(1) needs to be added to the bootstrap kit
+ # first if we want to use it here
+ #eval $(${STAT} -s ${f} 2>/dev/null)
+ echo "pkgcache_begin ${fn}" >> ${tmpd}/${cachefile}
+ #echo "pkgcache_mtime=${st_mtime}" >> ${tmpd}/${cachefile}
+ if test "X${DEBUG}" = "Xyes" ; then
+ echo "${PKG_INFO} -q -B ${f}"
+ fi
+ ${PKG_INFO} -q -B ${f} >> ${tmpd}/${cachefile}
+ echo "pkgcache_end ${fn}" >> ${tmpd}/${cachefile}
+ done
+ mv ${tmpd}/${cachefile} ${d}/${cachefile}
+ fi
+
+
+}
+
+process_cache_files(){
+ echo "${prompt}Checking master cache file ${PACKAGES}/${cachefile}"
+ echo "pkgcache_version ${cacheversion}" > ${tmpd}/${cachefile}
+ if test -n "${all_cache_files}" ; then
+ for c in ${all_cache_files} ; do
+ echo "pkgcache_cachefile ${c}" >> ${tmpd}/${cachefile}
+ done
+ fi
+ if test ! -f ${PACKAGES}/${cachefile} ; then
+ echo "${tab}Creating master cache file ${PACKAGES}/${cachefile}"
+ cp ${tmpd}/${cachefile} ${PACKAGES}/${cachefile}
+ fi
+ if ${CMP} -s ${tmpd}/${cachefile} ${PACKAGES}/${cachefile} ; then
+ echo "${tab}Master cache file ${PACKAGES}/${cachefile} is up to date"
+ else
+ echo "${tab}Updating master cache file ${PACKAGES}/${cachefile}"
+ mv ${tmpd}/${cachefile} ${PACKAGES}/${cachefile}
+ fi
+}
+
+######################################################################
+#
+# Handle command line options
+#
+######################################################################
+
+DEBUG=no
+verbose=no
+force=no
+
+while
Home |
Main Index |
Thread Index |
Old Index