Re: Replacing NFS-shared DISTDIR

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Thu, Sep 10, 2015 at 07:51:49AM +0200, Joerg Sonnenberger wrote:
> Hello all,
> a common problem in bulk builds using multiple chroots or even machines
> is the need to share distfiles. If you have ten chroots, you simply don't
> want to waste bandwidth or storage for ten copies of all the files. The
> normal solution to that was a shared filesystem, either null mounts
> (locally) or NFS (remotely). This works quite well once the cache has
> been populated, but it runs into problems when multiple instances tries
> to fetch the same distfile.

On the TNF bulk builders we have a NFS-shared $DISTFILES.
We use in mk.conf:
FETCH_USING=       custom
FETCH_CMD=         /root/dofetch
FETCH_BEFORE_ARGS=
FETCH_RESUME_ARGS= -R
FETCH_OUTPUT_ARGS= -o
FETCH_AFTER_ARGS= 

with /root/dofetch being:
#!/bin/sh -e
hostname=$(hostname -s)
wrkdir=/distfiles/${hostname}
oldpwd=$(pwd)

if [ ! -d ${wrkdir} ] ; then
        mkdir ${wrkdir}
fi

out="."
while getopts Ro: f ; do
        case $f in
        R) ;;
        o) out=$OPTARG;;
        \?) echo "usage: $0 [-R] [-o file] URL"; exit 1;;
        esac
done
shift `expr $OPTIND - 1`

cd ${wrkdir}
rm -f * || true
ftp $*
cd ${oldpwd}
mv ${wrkdir}/* ${out}
exit 0


It works quite well.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--