pkg/36774: pkg_install may prematurely abort file transfer

[he%uninett.no@localhost]

>Number:         36774
>Category:       pkg
>Synopsis:       pkg_install may prematurely abort file transfer
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 12 16:25:00 +0000 2007
>Originator:     Havard Eidnes
>Release:        NetBSD 3.0_STABLE
>Organization:
        I Try...
>Environment:
System: NetBSD stegg.urc.uninett.no 3.0_STABLE NetBSD 3.0_STABLE (STEGG.MP) #1: 
Mon Jul 10 16:13:40 CEST 2006 
he%stegg.urc.uninett.no@localhost:/usr/obj/sys/arch/i386/compile/STEGG.MP i386
Architecture: i386
Machine: i386
>Description:
        At home, accessing an FTP server over a wireless network
        with experienced maximum capacity in the area of 900Kbit/s,
        but sometimes significantly lower, a pkg_add of the
        openoffice2 package (implying a file transfer from the ftp
        server via PKG_PATH) would not complete, but rather be
        aborted by pkg_install because it thinks the file transfer
        took "too long".

        The reason appears to be that there is a constant time-out
        between pkg_add and the ftp sub-process, and there appears to
        be no progress detection for the ftp sub-process, so the
        constand time-out may fire even though the file transfer
        is still busy transferring the file for the package.

        This problem was experienced both with an older version of
        pkg_install as well as pkg_install-20070804.

        Here is the output from "pkg_add -v" for a failed attempt
        using the newer pkg_install:

stegg# pkg_add -v openoffice2
parsing: 
ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
path: 
ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
increasing RLIMIT_NOFILE to max. 1772 open files
trying PKG_PATH 
ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All
Spawning FTP coprocess
ftp -detv 
ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/

ftp> prompt off

ftp> nlist openoffice2 /var/tmp/pkg.21190a
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tbz /var/tmp/pkg.21190b
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tgz /var/tmp/pkg.21190c
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-*.t[bg]z /var/tmp/pkg.21190d

ftp> cd .
best match: 
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-[0-9]*.t[bg]z'
 expanded to 
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
Trying to fetch 
ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz.
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-2.2.1.tgz /var/tmp/pkg.21190f

ftp> cd .
best match: 
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
Reusing FDs 4/5 for communication to FTP coprocess
unpackURL 
'ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
 to '/var/tmp/instmp.21190e'

ftp> get openoffice2-2.2.1.tgz "| ( cd /var/tmp/instmp.21190e; tar 
--use-compress-program gzip -vvxp -f - | tee /dev/stderr )"
pkg_add: expect: poll() timeout

ftp> cd .
gzip: (stdin): unexpected end of file

tar: End of archive volume 1 reached
tar: ustar vol 1, 263 files, 143504894 bytes read, 0 bytes written in 601 secs 
(238776 bytes/sec)
+CONTENTS
+COMMENT
+DESC
+DISPLAY
+BUILD_VERSION
+BUILD_INFO
+SIZE_PKG
+SIZE_ALL
OpenOffice.org2.2.1/LICENSE
OpenOffice.org2.2.1/LICENSE.html
OpenOffice.org2.2.1/README
OpenOffice.org2.2.1/README.html
OpenOffice.org2.2.1/THIRDPARTYLICENSEREADME.html
...
OpenOffice.org2.2.1/program/libsvl680bi.so
OpenOffice.org2.2.1/program/libsvt680bi.so
OpenOffice.org2.2.1/program/libsvx680bi.sotar: Unexpected EOF on archive file
pkg_add: Cannot fetch file (-1!=226)!
Error on 
unpackURL('ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz',
 '/var/tmp/instmp.21190e')
pkg_add: unable to fetch 
`ftp://securitate.uninett.no/pub/NetBSD/packages/pkgsrc-2007Q2/NetBSD-2.1/i386/All/openoffice2-2.2.1.tgz'
 by URL
pkg_add: unable to open table of contents file `+CONTENTS' - not a package?
pkg_add: can't chdir back to '/var/tmp/instmp.21190e'
ftp: No control connection for command.
stegg# 

        Transferring the openoffice-2.2.1.tgz file manually got it
        transferred in 16:25 minutes:

ftp> get openoffice2-2.2.1.tgz
local: openoffice2-2.2.1.tgz remote: openoffice2-2.2.1.tgz
229 Entering Extended Passive Mode (|||59246|)
150 Opening BINARY mode data connection for 'openoffice2-2.2.1.tgz' (110846667 
bytes).
100% |*************************************|   105 MB  109.86 KB/s    00:00 ETA
226 Transfer complete.
110846667 bytes received in 16:25 (109.86 KB/s)
ftp> 

        while the only constant poll() timeout in pkg_install appears to
        specify 1 hour as the maximum to wait for ftp to transfer a file:

    rc = poll(set, 1, 60*60*1000);    /* seconds until next message from tar */

        I seemed to have the impression that pkg_add timed out before one
        hour had passed.

>How-To-Repeat:
        See above for conditions.
>Fix:
        Sorry, I don't know, but rather obviously using a constant
        time-out for the file transfer is bound to trigger issues
        such as this, so a better "progress" detector for the file
        transfer process is IMHO needed.