CVS commit: pkgsrc/www/curl

[Thomas Klausner <wiz%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   wiz
Date:           Tue Feb 16 12:51:44 UTC 2010

Modified Files:
        pkgsrc/www/curl: Makefile distinfo
Removed Files:
        pkgsrc/www/curl/patches: patch-ab

Log Message:
Update to 7.20.0:

Version 7.20.0 (9 February 2010)

Daniel Stenberg (9 Feb 2010)
- When downloading compressed content over HTTP and the app asked libcurl to
  automatically uncompress it with the CURLOPT_ENCODING option, libcurl could
  wrongly provide the callback with more data than the maximum documented
  amount. An application could thus get tricked into badness if the maximum
  limit was trusted to be enforced by libcurl itself (as it is documented).

  This is further detailed and explained in the libcurl security advisory
  20100209 at

    http://curl.haxx.se/docs/adv_20100209.html

Daniel Fandrich (3 Feb 2010)
- Changed the Watcom makefiles to make them easier to keep in sync with
  Makefile.inc since that can't be included directly.

Yang Tse (2 Feb 2010)
- Symbol CURL_FORMAT_OFF_T now obsoleted, will be removed in a future release,
  symbol will not be available when building with CURL_NO_OLDIES defined. Use
  of CURL_FORMAT_CURL_OFF_T is preferred since 7.19.0

Daniel Stenberg (1 Feb 2010)
- Using the multi_socket API, it turns out at times it seemed to "forget"
  connections (which caused a hang). It turned out to be an existing (7.19.7)
  bug in libcurl (that's been around for a long time) and it happened like
  this:

  The app calls curl_multi_add_handle() to add a new easy handle, libcurl will
  then set it to timeout in 1 millisecond so libcurl will tell the app about
  it.

  The app's timeout fires off that there's a timeout, the app calls libcurl as
  we so often document it:

  do {
   res = curl_multi_socket_action(... TIMEOUT ...);
  } while(CURLM_CALL_MULTI_PERFORM == res);

  And this is the problem number one:

  When curl_multi_socket_action() is called with no specific handle, but only
  a timeout-action, it will *only* perform actions within libcurl that are
  marked to run at this time. In this case, the request would go from INIT to
  CONNECT and return CURLM_CALL_MULTI_PERFORM. When the app then calls libcurl
  again, there's no timer set for this handle so it remains in the CONNECT
  state. The CONNECT state is a transitional state in libcurl so it reports no
  sockets there, and thus libcurl never tells the app anything more about that
  easy handle/connection.

  libcurl _does_ set a 1ms timeout for the handle at the end of
  multi_runsingle() if it returns CURLM_CALL_MULTI_PERFORM, but since the loop
  is instant the new job is not ready to run at that point (and there's no
  code that makes libcurl call the app to update the timout for this new
  timeout). It will simply rely on that some other timeout will trigger later
  on or that something else will update the timeout callback. This makes the
  bug fairly hard to repeat.

  The fix made to adress this issue:

  We introduce a loop in lib/multi.c around all calls to multi_runsingle() and
  simply check for CURLM_CALL_MULTI_PERFORM internally. This has the added
  benefit that this goes in line with my long-term wishes to get rid of the
  CURLM_CALL_MULTI_PERFORM all together from the public API.

  The downside of this fix, is that the counter we return in 'running_handles'
  in several of our public functions then gets a slightly new and possibly
  confusing behavior during times:

  If an app adds a handle that fails to connect (very quickly) it may just
  as well never appear as a 'running_handle' with this fix. Previously it
  would first bump the counter only to get it decreased again at next call.
  Even I have used that change in handle counter to signal "end of a
  transfer". The only *good* way to find the end of a individual transfer
  is calling curl_multi_info_read() to see if it returns one.

  Of course, if the app previously did the looping before it checked the
  counter, it really shouldn't be any new effect.

Yang Tse (26 Jan 2010)
- Constantine Sapuntzakis' and Joshua Kwan's work done in the last four months
  relative to the asynchronous DNS lookups, along with with some integration
  adjustments I have done are finally committed to CVS.

  Currently these enhancements will benefit builds done using c-ares on any
  platform as well as Windows builds using the default threaded resolver.

  This release does not make generally available POSIX threaded DNS lookups
  yet. There is no configure option to enable this feature yet. It is possible
  to experimantally try this feature running configure with compiler flags that
  make simultaneous definition of preprocessor symbols USE_THREADS_POSIX and
  HAVE_PTHREAD_H, as well as whatever reentrancy compiler flags and linker ones
  are required to link and properly use pthread_* functions on each platform.

Daniel Stenberg (26 Jan 2010)
- Mike Crowe made libcurl return CURLE_COULDNT_RESOLVE_PROXY when it is the
  proxy that cannot be resolved when using c-ares. This matches the behaviour
  when not using c-ares.

Bj
- Added a new flag: -J/--remote-header-name. This option tells the
  -O/--remote-name option to use the server-specified Content-Disposition
  filename instead of extracting a filename from the URL.

Daniel Stenberg (21 Jan 2010)
- Chris Conroy brought support for RTSP transfers, and with it comes 8(!) new
  libcurl options for controlling what to get and how to receive posssibly
  interleaved RTP data.

Daniel Stenberg (20 Jan 2010)
- As was pointed out on the http-state mailing list, the order of cookies in a
  HTTP Cookie: header _needs_ to be sorted on the path length in the cases
  where two cookies using the same name are set more than once using
  (overlapping) paths. Realizing this, identically named cookies must be
  sorted correctly. But detecting only identically named cookies and take care
  of them individually is harder than just to blindly and unconditionally sort
  all cookies based on their path lengths. All major browsers also already do
  this, so this makes our behavior one step closer to them in the cookie area.

  Test case 8 was the only one that broke due to this change and I updated it
  accordingly.

Daniel Stenberg (19 Jan 2010)
- David McCreedy brought a fix and a new test case (129) to make libcurl work
  again when downloading files over FTP using ASCII and it turns out that the
  final size of the file is not the same as the initial size the server
  reported. This is very common since servers don't take the newline
  conversions into account.

Kamil Dudka (14 Jan 2010)
- Suppressed side effect of OpenSSL configure checks, which prevented NSS from
  being properly detected under certain circumstances. It had been caused by
  strange behavior of pkg-config when handling PKG_CONFIG_LIBDIR. pkg-config
  distinguishes among empty and non-existent environment variable in that case.

Daniel Stenberg (12 Jan 2010)
- Gil Weber reported a peculiar flaw with the multi interface when doing SFTP
  transfers: curl_multi_fdset() would return -1 and not set and file
  descriptors several times during a transfer of a single file. It turned out
  to be due to two different flaws now fixed. Gil's excellent recipe helped me
  nail this.

Daniel Stenberg (11 Jan 2010)
- Made sure that the progress callback is repeatedly called at a regular
  interval even during very slow connects.

- The tests/runtests.pl script now checks to see if the test case that runs is
  present in the tests/data/Makefile.am and outputs a notice message on the
  screen if not. Each test file has to be included in that Makefile.am to get
  included in release archives and forgetting to add files there is a common
  mistake. This is an attempt to make it harder to forget.

Daniel Stenberg (9 Jan 2010)
- Johan van Selst found and fixed a OpenSSL session ref count leak:

  ossl_connect_step3() increments an SSL session handle reference counter on
  each call. When sessions are re-used this reference counter may be
  incremented many times, but it will be decremented only once when done (by
  Curl_ossl_session_free()); and the internal OpenSSL data will not be freed
  if this reference count remains positive. When a session is re-used the
  reference counter should be corrected by explicitly calling
  SSL_SESSION_free() after each consecutive SSL_get1_session() to avoid
  introducing a memory leak.

  (http://curl.haxx.se/bug/view.cgi?id=2926284)

Daniel Stenberg (7 Jan 2010)
- Make sure the progress callback is called repeatedly even during very slow
  name resolves when c-ares is used for resolving.

Claes Jakobsson (6 Jan 2010)
- Julien Chaffraix fixed so that the fragment part in an URL is not sent
  to the server anymore.

Kamil Dudka (3 Jan 2010)
- Julien Chaffraix eliminated a duplicated initialization in singlesocket().

Daniel Stenberg (2 Jan 2010)
- Make curl support --ssl and --ssl-reqd instead of the previous FTP-specific
  versions --ftp-ssl and --ftp-ssl-reqd as these options are now used to
  control SSL/TLS for IMAP, POP3 and SMTP as well in addition to FTP. The old
  option names are still working but the new ones are the ones listed and
  documented.

Daniel Stenberg (1 Jan 2010)
- Ingmar Runge enhanced libcurl's FTP engine to support the PRET command. This
  command is a special "hack" used by the drftpd server, but even though it is
  a custom extension I've deemed it fine to add to libcurl since this server
  seems to survive and people keep using it and want libcurl to support
  it. The new libcurl option is named CURLOPT_FTP_USE_PRET, and it is also
  usable from the curl tool with --ftp-pret. Using this option on a server
  that doesn't support this command will make libcurl fail.

  I added test cases 1107 and 1108 to verify the functionality.

  The PRET command is documented at
  http://www.drftpd.org/index.php/Distributed_PASV

Yang Tse (30 Dec 2009)
- Steven M. Schweda improved VMS build system, and Craig A. Berry helped
  with the patch and testing.

Daniel Stenberg (26 Dec 2009)
- Renato Botelho and Peter Pentchev brought a patch that makes the libcurl
  headers work correctly even on FreeBSD systems before v8.

  (http://curl.haxx.se/bug/view.cgi?id=2916915)

Daniel Stenberg (17 Dec 2009)
- David Byron fixed Curl_ossl_cleanup to actually call ENGINE_cleanup when
  available.

- Follow-up fix for the proxy fix I did for Jon Nelson's bug. It turned out I
  was a bit too quick and broke test case 1101 with that change. The order of
  some of the setups is sensitive. I now changed it slightly again to make
  sure we do them in this order:

  1 - parse URL and figure out what protocol is used in the URL
  2 - prepend protocol:// to URL if missing
  3 - parse name+password off URL, which needs to know what protocol is used
      (since only some allows for name+password in the URL)
  4 - figure out if a proxy should be used set by an option
  5 - if no proxy option, check proxy environment variables
  6 - run the protocol-specific setup function, which needs to have the proxy
      already set

Daniel Stenberg (15 Dec 2009)
- Jon Nelson found a regression that turned out to be a flaw in how libcurl
  detects and uses proxies based on the environment variables. If the proxy
  was given as an explicit option it worked, but due to the setup order
  mistake proxies would not be used fine for a few protocols when picked up
  from '[protocol]_proxy'. Obviously this broke after 7.19.4. I now also added
  test case 1106 that verifies this functionality.

  (http://curl.haxx.se/bug/view.cgi?id=2913886)

Daniel Stenberg (12 Dec 2009)
- IMAP, POP3 and SMTP support and their TLS versions (including IMAPS, POP3S
  and SMTPS) are now supported. The current state may not yet be solid, but
  the foundation is in place and the test suite has some initial support for
  these protocols. Work will now persue to make them nice libcurl citizens
  until release.

  The work with supporting these new protocols was sponsored by
  networking4all.com - thanks!

Daniel Stenberg (10 Dec 2009)
- Siegfried Gyuricsko found out that the curl manual said --retry would retry
  on FTP errors in the transient 5xx range. Transient FTP errors are in the
  4xx range. The code itself only tried on 5xx errors that occured _at login_.
  Now the retry code retries on all FTP transfer failures that ended with a
  4xx response.

  (http://curl.haxx.se/bug/view.cgi?id=2911279)

- Constantine Sapuntzakis figured out a case which would lead to libcurl
  accessing alredy freed memory and thus crash when using HTTPS (with
  OpenSSL), multi interface and the CURLOPT_DEBUGFUNCTION and a certain order
  of cleaning things up. I fixed it.

  (http://curl.haxx.se/bug/view.cgi?id=2905220)

Daniel Stenberg (7 Dec 2009)
- Martin Storsjo made libcurl use the Expect: 100-continue header for posts
  with unknown size. Previously it was only used for posts with a known size
  larger than 1024 bytes.

Daniel Stenberg (1 Dec 2009)
- If the Expect: 100-continue header has been set by the application through
  curl_easy_setopt with CURLOPT_HTTPHEADER, the library should set
  data->state.expect100header accordingly - the current code (in 7.19.7 at
  least) doesn't handle this properly. Martin Storsjo provided the fix!

Yang Tse (28 Nov 2009)
- Added Diffie-Hellman parameters to several test harness certificate files in
  PEM format. Required by several stunnel versions used by our test harness.

Daniel Stenberg (28 Nov 2009)
- Markus Koetter provided a polished and updated version of Chad Monroe's TFTP
  rework patch that now integrates TFTP properly into libcurl so that it can
  be used non-blocking with the multi interface and more. BLKSIZE also works.

  The --tftp-blksize option was added to allow setting the TFTP BLKSIZE from
  the command line.

Daniel Stenberg (26 Nov 2009)
- Extended and fixed the change I did on Dec 11 for the the progress
  meter/callback during FTP command/response sequences. It turned out it was
  really lame before and now the progress meter SHOULD get called at least
  once per second.

Daniel Stenberg (23 Nov 2009)
- Bjorn Augustsson reported a bug which made curl not report any problems even
  though it failed to write a very small download to disk (done in a single
  fwrite call). It turned out to be because fwrite() returned success, but
  there was insufficient error-checking for the fclose() call which tricked
  curl to believe things were fine.

Yang Tse (23 Nov 2009)
- David Byron modified Makefile.dist vc8 and vc9 targets in order to allow
  finer granularity control when generating src and lib makefiles.

Yang Tse (22 Nov 2009)
- I modified configure to force removal of the curlbuild.h file included in
  distribution tarballs for use by non-configure systems. As intended, this
  would get overwriten when doing in-tree builds. But VPATH builds would end
  having two curlbuild.h files, one in the source tree and another in the
  build tree. With the modification I introduced 5 Nov 2009 this could become
  an issue when running libcurl's test suite.

Daniel Stenberg (20 Nov 2009)
- Constantine Sapuntzakis identified a write after close, as the sockets were
  closed by libcurl before the SSL lib were shutdown and they may write to its
  socket. Detected to at least happen with OpenSSL builds.

- Jad Chamcham pointed out a bug with connection re-use. If a connection had
  CURLOPT_HTTPPROXYTUNNEL enabled over a proxy, a subsequent request using the
  same proxy with the tunnel option disabled would still wrongly re-use that
  previous connection and the outcome would only be badness.

Yang Tse (18 Nov 2009)
- I modified the memory tracking system to make it intolerant with zero sized
  malloc(), calloc() and realloc() function calls.

Daniel Stenberg (17 Nov 2009)
- Constantine Sapuntzakis provided another fix for the DNS cache that could
  end up with entries that wouldn't time-out:

  1. Set up a first web server that redirects (307) to a http://server:port
     that's down
  2. Have curl connect to the first web server using curl multi

  After the curl_easy_cleanup call, there will be curl dns entries hanging
  around with in_use != 0.

  (http://curl.haxx.se/bug/view.cgi?id=2891591)

- Marc Kleine-Budde fixed: curl saved the LDFLAGS set during configure into
  its pkg-config file.  So -Wl stuff ended up in the .pc file, which is really
  bad, and breaks if there are multiple -Wl in our LDFLAGS (which are in
  PTXdist). bug #2893592 (http://curl.haxx.se/bug/view.cgi?id=2893592)

Kamil Dudka (15 Nov 2009)
- David Byron improved the configure script to use pkg-config to find OpenSSL
  (and in particular the list of required libraries) even if a path is given
  as argument to --with-ssl

Yang Tse (15 Nov 2009)
- I removed enable-thread / disable-thread configure option. These were only
  placebo options. The library is always built as thread safe as possible on
  every system.

Claes Jakobsson (14 Nov 2009)
- curl-config now accepts '--configure' to see what arguments was
  passed to the configure script when building curl.

Daniel Stenberg (14 Nov 2009)
- Claes Jakobsson restored the configure functionality to detect NSS when
  --with-nss is set but not "yes".

  I think we can still improve that to check for pkg-config in that path etc,
  but at least this patch brings back the same functionality we had before.

- Camille Moncelier added support for the file type SSL_FILETYPE_ENGINE for
  the client certificate. It also disable the key name test as some engines
  can select a private key/cert automatically (When there is only one key
  and/or certificate on the hardware device used by the engine)

Yang Tse (14 Nov 2009)
- Constantine Sapuntzakis provided the fix that ensures that an SSL connection
  won't be reused unless protection level for peer and host verification match.

  I refactored how preprocessor symbol _THREAD_SAFE definition is done.

Kamil Dudka (12 Nov 2009)
- Kevin Baughman provided a fix preventing libcurl-NSS from crash on doubly
  closed NSPR descriptor. The issue was hard to find, reported several times
  before and always closed unresolved. More info at the RH bug:
  https://bugzilla.redhat.com/534176

- libcurl-NSS now tries to reconnect with TLS disabled in case it detects
  a broken TLS server. However it does not happen if SSL version is selected
  manually. The approach was originally taken from PSM. Kaspar Brand helped me
  to complete the patch. Original bug reports:
  https://bugzilla.redhat.com/525496
  https://bugzilla.redhat.com/527771

Yang Tse (12 Nov 2009)
- I modified configure script to make the getaddrinfo function check also
  verify if the function is thread safe.

Yang Tse (11 Nov 2009)
- Marco Maggi reported that compilation failed when configured --with-gssapi
  and GNU GSS installed due to a missing mutual exclusion of header files in
  the Kerberos 5 code path. He also verified that my patch worked for him.

Daniel Stenberg (11 Nov 2009)
- Constantine Sapuntzakis posted bug #2891595
  (http://curl.haxx.se/bug/view.cgi?id=2891595) which identified how an entry
  in the DNS cache would linger too long if the request that added it was in
  use that long. He also provided the patch that now makes libcurl capable of
  still doing a request while the DNS hash entry may get timed out.

- Christian Schmitz noticed that the progress meter/callback was not properly
  used during the FTP connection phase (after the actual TCP connect), while
  it of course should be. I also made the speed check get called correctly so
  that really slow servers will trigger that properly too.

Kamil Dudka (5 Nov 2009)
- Dropped misleading timeouts in libcurl-NSS and made sure the SSL socket works
  in non-blocking mode.

Yang Tse (5 Nov 2009)
- I removed leading 'curl' path on the 'curlbuild.h' include statement in
  curl.h, adjusting auto-makefiles include path, to enhance portability to
  OS's without an orthogonal directory tree structure such as OS/400.

Daniel Stenberg (4 Nov 2009)
- I fixed several problems with the transfer progress meter. It showed the
  wrong percentage for small files, most notable for <1000 bytes and could
  easily end up showing more than 100% at the end. It also didn't show any
  percentage, transfer size or estimated transfer times when transferring
  less than 100 bytes.


To generate a diff of this commit:
cvs rdiff -u -r1.95 -r1.96 pkgsrc/www/curl/Makefile
cvs rdiff -u -r1.63 -r1.64 pkgsrc/www/curl/distinfo
cvs rdiff -u -r1.12 -r0 pkgsrc/www/curl/patches/patch-ab

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.