>Number:         28413
>Category:       bin
>Synopsis:       pkg_add can't handle simple ftp URLs
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Nov 24 18:30:00 +0000 2004
>Originator:     Robert Elz
>Release:        NetBSD 2.0B  (-- irrelevant, old bug, still in -current)
>Organization:
	Prince of Songkla University
>Environment:
System: NetBSD delta.noi.kre.to 2.0B NetBSD 2.0B (DELTA) #0: Fri Apr 16 23:25:38 ICT 2004 kre@lilac.noi.kre.to:/usr/obj/current/sys/arch/i386/compile/DELTA i386
Architecture: i386
Machine: i386
>Description:
	pkg_add cannot handle fetching a FTP URL of the form
		ftp://host/file.tgz
	That is, where the file is present in the ftp login
	directory on the server host.

	Aside: I have entered this PR in the bin category, as
	pkg_add libes in /usr/src/usr.sbin/pkg_install/*
	but it could also be in pkg I guess, given that it
	also lives in pkgsrc/pkgtools/pkg_install.  I don't
	think it is possible to have a PR in 2 categories is it?
	(I'm not wasting time here testing it).

	The bug probably affects all of the pkg_* commands that
	can take a ftp:// type URL (as they all share common code).

>How-To-Repeat:
	Just do
		pkg_add ftp://hostname/anything.tgz
	"hostname" must be a valid name of a host with anon FTP
	enabled, but that's all that's needed to demonstrate the
	problem, "anything.tgz" doesn't need to be a pkg name,
	and what's more, doesn't need to exist.

	If you're prepared to wait long enough (I think it is 10
	minutes) I believe this is supposed to correct itself,
	and the pkg_add will abort (ie: it won't work, but it
	shou;dn't hang forever either) but I've never had the
	patience to wait that long to be sure that works.

>Fix:
	No fix, but I can explain the problem.

	First, the base cause is the absurd attempt to optimise FTP
	connections.   Even at its worst, pkg_add isn't going to need
	to make more than 15-20 fetches to get all of the bits needed
	to install a package (and if there are a few worse cases,
	that I've never seen, who cares anyway).   Simply using one
	ftp command at a time, for each file required, giving ftp the
	URL of the file, and waiting for FTP to finish fetching it
	would be the obvious, simple, and clean way to handle this,
	and if that means a few extra secods of waiting for the RTTs
	needed to connect to the server several times, then so what?

	But that's not what is done - instead an attempt is made to
	open one connection, and then use it over and over again to
	get all the files that are required (dependencies and all
	that).

	To do that, pkg_add runs ftp with a URL that is of the
	directory containing the file (ie: the URL minus the
	last component), expecting the ftp process to use the
	FTP CWD command to enter the directory it assumes will
	be in the URL.   It then waits for a "success" reply to
	the CWD command it is expecting will be issued using this
	piece of code ...

                if ((expect(ftpio.answer, "\n(221|250|221|550).*\n", &rc) != 0)
                    || rc != 250) {
                        warnx("expect1 failed, rc=%d", rc);
                        return -1;
                }

	That's from pkg_install/lib/ftpio.c in ftp_start().

	Why 2 different 221's are needed I'm not going to guess at,
	but that's not the cause of the problem, the problem is
	in expecting to get a "250" response to the opening
	sequence.    That's what happens when a CWD is issued,
	but without one of those, there is no 250, and the expect()
	call just sits and waits (until it times out).

	Possible fixes...

	Force the URL to have a trailing "./" appended to it.
		That will ensure that a CWD command is always issued
		and so a 250 response can be expected.   Of course,
		it also assumes a unix filename compatible server is
		being used (but is there anything else any more?)
	Just wait for the FTP login response, instead of (one of the)
		CWD command response, 230 would probably work.
		This is OK, because there can be many 250 responses,
		pkg_add sends a "prompt off" command after the login
		sequence, so it can synchronise itself with the server.
		It does assume that all servers correctly reply 230
		to a successful login.
	Get rid of the "just one ftp" nonsense, and simplify everything.
		(major change).   This one is certainly safe (and would
		mean that ftp debug mode is no longer needed, allow the
		possibility of using different ftp commands, and probably
		lots of other wins).

	And perhaps others.

ps: there's no need to fix this before 2.0, though the bug
is certainly present in its RCn versions (that's where I
noticed it first).   Fetching packages directly from the
ftp login directory seems to be something perverse enough
that I'm the only one to have ever done it (ugh - attempted it,
since it failed.)