>Number:         36774
>Category:       pkg
>Synopsis:       pkg_install may prematurely abort file transfer
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 12 16:25:00 +0000 2007
>Originator:     Havard Eidnes
>Release:        NetBSD 3.0_STABLE
>Organization:
	I Try...
>Environment:
System: NetBSD stegg.urc.uninett.no 3.0_STABLE NetBSD 3.0_STABLE (STEGG.MP) #1: Mon Jul 10 16:13:40 CEST 2006 he@stegg.urc.uninett.no:/usr/obj/sys/arch/i386/compile/STEGG.MP i386
Architecture: i386
Machine: i386
>Description:
	At home, accessing an FTP server over a wireless network
	with experienced maximum capacity in the area of 900Kbit/s,
	but sometimes significantly lower, a pkg_add of the
	openoffice2 package (implying a file transfer from the ftp
	server via PKG_PATH) would not complete, but rather be
	aborted by pkg_install because it thinks the file transfer
	took "too long".

	The reason appears to be that there is a constant time-out
	between pkg_add and the ftp sub-process, and there appears to
	be no progress detection for the ftp sub-process, so the
	constand time-out may fire even though the file transfer
	is still busy transferring the file for the package.

	This problem was experienced both with an older version of
	pkg_install as well as pkg_install-20070804.

	Here is the output from "pkg_add -v" for a failed attempt
	using the newer pkg_install:

stegg# pkg_add -v openoffice2
parsing: ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
path: ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
increasing RLIMIT_NOFILE to max. 1772 open files
trying PKG_PATH ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
Spawning FTP coprocess
ftp -detv ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/

ftp> prompt off

ftp> nlist openoffice2 /var/tmp/pkg.21190a
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tbz /var/tmp/pkg.21190b
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tgz /var/tmp/pkg.21190c
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-*.t[bg]z /var/tmp/pkg.21190d

ftp> cd .
best match: 'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-[0-9]*.t[bg]z' expanded to 'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
Trying to fetch ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz.
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-2.2.1.tgz /var/tmp/pkg.21190f

ftp> cd .
best match: 'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
Reusing FDs 4/5 for communication to FTP coprocess
unpackURL 'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz' to '/var/tmp/instmp.21190e'

ftp> get openoffice2-2.2.1.tgz "| ( cd /var/tmp/instmp.21190e; tar --use-compress-program gzip -vvxp -f - | tee /dev/stderr )"
pkg_add: expect: poll() timeout

ftp> cd .
gzip: (stdin): unexpected end of file

tar: End of archive volume 1 reached
tar: ustar vol 1, 263 files, 143504894 bytes read, 0 bytes written in 601 secs (238776 bytes/sec)
+CONTENTS
+COMMENT
+DESC
+DISPLAY
+BUILD_VERSION
+BUILD_INFO
+SIZE_PKG
+SIZE_ALL
OpenOffice.org2.2.1/LICENSE
OpenOffice.org2.2.1/LICENSE.html
OpenOffice.org2.2.1/README
OpenOffice.org2.2.1/README.html
OpenOffice.org2.2.1/THIRDPARTYLICENSEREADME.html
...
OpenOffice.org2.2.1/program/libsvl680bi.so
OpenOffice.org2.2.1/program/libsvt680bi.so
OpenOffice.org2.2.1/program/libsvx680bi.sotar: Unexpected EOF on archive file
pkg_add: Cannot fetch file (-1!=226)!
Error on unpackURL('ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz', '/var/tmp/instmp.21190e')
pkg_add: unable to fetch `ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz' by URL
pkg_add: unable to open table of contents file `+CONTENTS' - not a package?
pkg_add: can't chdir back to '/var/tmp/instmp.21190e'
ftp: No control connection for command.
stegg# 

	Transferring the openoffice-2.2.1.tgz file manually got it
	transferred in 16:25 minutes:

ftp> get openoffice2-2.2.1.tgz
local: openoffice2-2.2.1.tgz remote: openoffice2-2.2.1.tgz
229 Entering Extended Passive Mode (|||59246|)
150 Opening BINARY mode data connection for 'openoffice2-2.2.1.tgz' (110846667 bytes).
100% |*************************************|   105 MB  109.86 KB/s    00:00 ETA
226 Transfer complete.
110846667 bytes received in 16:25 (109.86 KB/s)
ftp> 

	while the only constant poll() timeout in pkg_install appears to
	specify 1 hour as the maximum to wait for ftp to transfer a file:

    rc = poll(set, 1, 60*60*1000);    /* seconds until next message from tar */

	I seemed to have the impression that pkg_add timed out before one
	hour had passed.

>How-To-Repeat:
	See above for conditions.
>Fix:
	Sorry, I don't know, but rather obviously using a constant
	time-out for the file transfer is bound to trigger issues
	such as this, so a better "progress" detector for the file
	transfer process is IMHO needed.