pkgsrc issues fetching source on HP-UX

[Xavier Nye <xaviernye%msn.com@localhost>]

This is a followup to my original question of "Force pkgsrc to use IPv4 when downloading package source?" However because it has been a couple of days since that conversation because I've done some more debugging, and I don't think the actual issue relates to what the title is, I'm assuming it's best to just start a new thread about this rather than replying to the original.

The original error was using the system ftp. I ended up building pkgsrc's tnftp to see if it would work any better,but at this point I've basically given up on tnftp because even when I configure it using some configuration of -inet6 and -ssl it still refuses to work. For anyone who's interested, here's the output for -inet6 -ssl

bash-4.3$ bmake

=> Bootstrap dependency digest>=20010302: found digest-20190127

=> Bootstrap dependency tnftp-[0-9]*: found tnftp-20151004nb2

=> Fetching wget-1.20.3.tar.lz

=> Total size: 2135272 bytes

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unsupported URL scheme `https'

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unsupported URL scheme `https'

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

Trying 151.101.1.6:80 ...

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Can't LOOKUP `ftp6.NetBSD.org:http': no address associated with host

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

ftp: Unable to determine rcvbuf size: Bad address

fetch: Unable to fetch expected file wget-1.20.3.tar.lz

*** Error code 1

Stop.

bmake: stopped in /home/build/pkgsrc/net/wget

> I also see errors about ftp not understanding https address. Maybe using:

> FETCH_USING?=wget
> or

> FETCH_USING?= curl

I've tried both wget and curl at this point. Here's my latest mk.conf using curl.

# Example /usr/pkg/etc/mk.conf file produced by bootstrap-pkgsrc

# Mon Aug  3 16:08:17 MDT 2020

.ifdef BSD_PKG_MK       # begin pkgsrc settings

ABI=                    64

PKG_DBDIR=              /usr/pkg/pkgdb

LOCALBASE=              /usr/pkg

VARBASE=                /var

PKG_TOOLS_BIN=          /usr/pkg/sbin

PKGINFODIR=             info

PKGMANDIR=              man

FETCH_USING=            curl

TOOLS_PLATFORM.install?=        /usr/pkg/bin/install-sh

TOOLS_PLATFORM.awk?=            /usr/pkg/bin/nawk

TOOLS_PLATFORM.sed?=            /usr/pkg/bin/nbsed

TOOLS_PLATFORM.fetch?=          /usr/local/bin/curl

.endif                  # end pkgsrc settings

One minor problem is that pkgsrc seems to want to use the native wget or curl and tries to pull that in. This shouldn't be a huge problem because I could just build that. However, this is where the real error seems to begin. For whatever reason pkgsrc seems to run into issues with the system /usr/bin/env when attempting to fetch sources. Here's a debug output of bmake -d e using curl to fetch sources.

bash-4.3# pwd

/home/build/pkgsrc/devel/flex

bash-4.3# clear

bash-4.3# pwd

/home/build/pkgsrc/devel/flex

bash-4.3# bmake -d e

=> Bootstrap dependency curl-[0-9]*: NOT found

=> Verifying reinstall for ../../www/curl

=> Bootstrap dependency digest>=20010302: found digest-20190127

=> Fetching curl-7.71.1.tar.xz

=> Total size: 2387660 bytes

/usr/bin/env: illegal option -- -

/usr/bin/env: illegal option -- f

/usr/bin/env: illegal option -- a

/usr/bin/env: illegal option -- l

/usr/bin/env: illegal option -- -

/usr/bin/env: illegal option -- n

/usr/bin/env: illegal option -- s

/usr/bin/env: illegal option -- e

/usr/bin/env: illegal option -- c

/usr/bin/env: illegal option -- u

/usr/bin/env: illegal option -- r

/usr/bin/env: illegal option -- e

/usr/bin/env: illegal option -- -

/usr/bin/env: illegal option -- l

/usr/bin/env: illegal option -- o

/usr/bin/env: illegal option -- c

/usr/bin/env: illegal option -- a

/usr/bin/env: illegal option -- t

/usr/bin/env: illegal option -- o

/usr/bin/env: illegal option -- n

/usr/bin/env: illegal option -- -

/usr/bin/env: illegal option -- r

/usr/bin/env: illegal option -- e

/usr/bin/env: illegal option -- m

/usr/bin/env: illegal option -- o

/usr/bin/env: illegal option -- t

/usr/bin/env: illegal option -- e

/usr/bin/env: illegal option -- -

/usr/bin/env: illegal option -- t

/usr/bin/env: illegal option -- m

/usr/bin/env: illegal option -- e

/usr/bin/env: illegal option -- o

usage: env [-] [-i] [name=value] ... [utility [argument ...]]

fetch: Unable to fetch expected file curl-7.71.1.tar.xz

...

*** Failed target:  /home/build/pkgsrc/distfiles/curl-7.71.1.tar.xz

*** Failed command: set -e; unsorted_sites="https://curl.haxx.se/download/"; sites=" `/usr/bin/echo $unsorted_sites | /usr/pkg/bin/nawk -v seed=$$ 'BEGIN { srand(seed); ORS = \" \" } { n = split($0, site); for (i = n; i > 0; i--) { ir = int(rand() * i + 1); t = site[i]; site[i] = site[ir]; site[ir] = t; print site[i]; } }'` http://cdn.NetBSD.org/pub/pkgsrc/distfiles/ http://ftp6.NetBSD.org/pub/pkgsrc/distfiles/ http://ftp.fr.NetBSD.org/pub/pkgsrc/distfiles/ http://ftp.NetBSD.org/pub/pkgsrc/distfiles/"; cd /home/build/pkgsrc/distfiles && /usr/bin/env CHECKSUM=/usr/bin/env\ DIGEST=/usr/pkg/bin/digest\ CAT=/bin/cat\ \ ECHO=/usr/bin/echo\ SED=/usr/pkg/bin/nbsed\ \ \ TEST=test\ \ \ \ \ \ /bin/sh\ /home/build/pkgsrc/mk/checksum/checksum CP=/bin/cp ECHO=/usr/bin/echo FETCH_CMD=/usr/bin/env\ \ \ \ \ \ \ \ FETCH_BEFORE_ARGS=\ \ --fail\ --insecure\ --location\ --remote-time\ \ \ \ \ \ \ \ FETCH_AFTER_ARGS=-O FETCH_RESUME_ARGS=-C\ - FETCH_OUTPUT_ARGS=-o MKDIR=/bin/mkdir MV=/bin/mv RM=/bin/rm TEST=test TOUCH=/usr/bin/touch WC=/usr/bin/wc /bin/sh /home/build/pkgsrc/mk/fetch/fetch -f /home/build/pkgsrc/www/curl/distinfo curl-7.71.1.tar.xz $sites

*** Error code 1

Stop.

bmake[1]: stopped in /home/build/pkgsrc/www/curl

*** Failed target:  _pkgformat-bootstrap-depends

*** Failed command: set -e;/usr/bin/env AWK=/usr/pkg/bin/nawk PKG_ADMIN=/usr/pkg/sbin/pkg_admin\ -K\ /usr/pkg/pkgdb PKGSRCDIR=/home/build/pkgsrc PWD_CMD=/bin/pwd SED=/usr/pkg/bin/nbsed /bin/sh /home/build/pkgsrc/mk/pkgformat/pkg/list-dependencies " "\ \ curl-\[0-9\]\*:../../www/curl\ digest\>=20010302:../../pkgtools/digest " " " " " " " " | while read type pattern dir; do test "$type" = "bootstrap" || continue; case $type in bootstrap) Type=Bootstrap;; tool) Type=Tool;; build) Type=Build;; test) Type=Test;; full) Type=Full;; esac; case $type in bootstrap|tool) if expr "NO" : '[yY][eE][sS]' >/dev/null; then extradep=""; else extradep=" flex-2.6.4"; fi; cross=no; archopt=TARGET_ARCH=hppa64; pkg=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -E "$pattern" || true`; ;; build|test|full) extradep=" flex-2.6.4"; cross=NO; archopt=; pkg=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -E "$pattern" || true`; ;; esac; case "$pkg" in "") /usr/bin/echo "=>" "$Type dependency $pattern: NOT found"; target=reinstall; /usr/bin/echo "=>" "Verifying $target for $dir"; [ -d "$dir" ] || /bin/sh /home/build/pkgsrc/mk/scripts/fail /usr/bin/echo 1>&2 "ERROR:" "[depends.mk] The directory \`\`$dir'' does not exist."; cd $dir; /usr/bin/env PATH=/home/build/pkgsrc/devel/flex/work/.wrapper/bin:/home/build/pkgsrc/devel/flex/work/.buildlink/bin:/home/build/pkgsrc/devel/flex/work/.gcc/bin:/home/build/pkgsrc/devel/flex/work/.tools/bin:/usr/pkg/bin:/usr/pkg/bin:/usr/local/coreutils/bin:/usr/local/bin:/usr/sbin:/opt/softbench/bin:/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/ansic/bin:/opt/mpi/bin:/opt/hparray/bin:/opt/nettladm/bin:/opt/upgrade/bin:/opt/fcms/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/graphics/common/bin:/opt/pd/bin:/opt/resmon/bin:/opt/gnome/bin:/opt/wbem/bin:/opt/wbem/sbin:/opt/firefox:/opt/mozilla:/opt/netscape:/opt/atok/bin:/opt/egb/bin:/opt/graphics/phigs/bin:/usr/sbin/diag/contrib:/opt/mx/bin:/opt/OpenSource/bin:/opt/vje/bin:/opt/perl/bin:/opt/java1.4/jre/bin:/opt/spb/bin:/opt/aCC/bin:/opt/cadvise/bin:/opt/prm/bin:/opt/langtools/bin:/opt/imake/bin:/opt/fortran90/bin:/opt/fortran90/contrib/bin:/opt/pascal/bin:/opt/ssh/bin:/sbin:/home/root _PKGSRC_DEPS="$extradep" PKGNAME_REQD="$pattern" USE_CROSS_COMPILE=$cross $archopt /usr/pkg/bin/bmake _MAKE=/usr/pkg/bin/bmake OPSYS=HPUX OS_VERSION=11.11 LOWER_OPSYS=hpux _PKGSRCDIR=/home/build/pkgsrc PKGTOOLS_VERSION=20200701 _CC=/opt/hp-gcc64/bin/gcc _PATH_ORIG=/usr/pkg/bin:/usr/local/coreutils/bin:/usr/local/bin:/usr/sbin:/opt/softbench/bin:/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/ansic/bin:/opt/mpi/bin:/opt/hparray/bin:/opt/nettladm/bin:/opt/upgrade/bin:/opt/fcms/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/graphics/common/bin:/opt/pd/bin:/opt/resmon/bin:/opt/gnome/bin:/opt/wbem/bin:/opt/wbem/sbin:/opt/firefox:/opt/mozilla:/opt/netscape:/opt/atok/bin:/opt/egb/bin:/opt/graphics/phigs/bin:/usr/sbin/diag/contrib:/opt/mx/bin:/opt/OpenSource/bin:/opt/vje/bin:/opt/perl/bin:/opt/java1.4/jre/bin:/opt/spb/bin:/opt/aCC/bin:/opt/cadvise/bin:/opt/prm/bin:/opt/langtools/bin:/opt/imake/bin:/opt/fortran90/bin:/opt/fortran90/contrib/bin:/opt/pascal/bin:/opt/ssh/bin:/sbin:/home/root _AUTOMATIC=yes $target; case $type in bootstrap|tool) pkg=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -E "$pattern" || true`;; build|test|full) pkg=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -E "$pattern" || true`;; esac; case "$pkg" in "") /usr/bin/echo 1>&2 "ERROR:" "[depends.mk] A package matching \`\`$pattern'' should"; /usr/bin/echo 1>&2 "ERROR:" " be installed, but one cannot be found. Perhaps there is a"; /usr/bin/echo 1>&2 "ERROR:" " stale work directory for $dir?"; exit 1; esac; /usr/bin/echo "=>" "Returning to build of flex-2.6.4"; ;; *) case $type in bootstrap|tool) objfmt=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -Q OBJECT_FMT "$pkg"`;; build|test|full) objfmt=`/usr/pkg/sbin/pkg_info -K /usr/pkg/pkgdb -Q OBJECT_FMT "$pkg"`;; esac; case "$objfmt" in "") /usr/bin/echo 1>&2 "WARNING:" "[depends.mk] Unknown object format for installed package $pkg" ;; SOM) ;; *) /usr/bin/echo 1>&2 "ERROR:" "[depends.mk] Installed package $pkg has an"; /usr/bin/echo 1>&2 "ERROR:" " object format \`\`$objfmt'' which differs from \`\`SOM''. Please"; /usr/bin/echo 1>&2 "ERROR:" " update the $pkg package to SOM."; exit 1; ;; esac; silent=; if test -z "${silent}"; then /usr/bin/echo "=>" "$Type dependency $pattern: found $pkg"; fi; ;; esac; done

*** Error code 1

Stop.

bmake: stopped in /home/build/pkgsrc/devel/flex

The env errors spell out the default FETCH_BEFORE_ARGS for curl (minus the "i"s because those are valid flags it seems), which are FETCH_BEFORE_ARGS=--fail --insecure --location --remote-time. For using wget, the same exact thing happens, except it spells out --no-check-certificate. I've also tried using the env from GNU Coreutils, and it still produces the same "illegal option" error, except that instead of multiple lines, it treats it all as a single arg and prints something similar to "env: illegal option -- -no-check-env"

Not sure what's causing this. Any ideas on what it is or how to fix it?