Re: Why SFTP performance sucks, and how to fix it

[pgut001%cs.auckland.ac.nz@localhost (Peter Gutmann)]

Martin Pool <mbp%samba.org@localhost> writes:
>On  8 Jul 2003, Peter Gutmann <pgut001%cs.auckland.ac.nz@localhost> wrote:
>>Another thing to keep in mind here is that if you signal a read/write of the
>>entire file at once, the receiver knows how much disk space it needs and can
>>lock the space before starting the receive
>
>I don't know of a means for a Unix or Windows server to "lock" disk space in
>that way.  I suppose the server might zero-fill the blocks between getting
>the start of the request and reading the bulk of it, but that seems highly
>contrived.

It's going to be system-specific, but is available on many systems.  For
example under Windows you can do something called "ghost locking" (which dates
back to early MSDOS days, it may have other names but that's what I heard
database developers call it) where you seek past the end of file and drop some
bytes there, which locks that amount of disk space.  The DOS-compatibility may
be why under Windows you have to explicitly ask for sparse files, and they can
only exist under NTFS5 (if you copy one of these things to a non-NTFS5 volume,
it expands out to its full size).  Various Unix systems can do this too
(either with a simple lseek or with an lssek and write), or have some
equivalent way to do it, even if it means shelling out to mkfile to do the
job.

In any event you don't *have* to do this, but if I'm copying a large amount of
data to an OS that can easily do the check and it doesn't, and then fails
halfway through the transfer, I won't be too happy with the software (or its
author :-).  Same with other user-friendliness options, if the client and
server can exchange hints at the start and the server can tell me in advance
of problems, I'll be a lot happier than having it fail at the end of a 500MB
transfer.

>Here's another example, taken from experience with Samba: if you're going to
>send chunks of 2GB at a time they presumably have to be "streamed" from disk
>and not built in a temporary buffer.

Hmm, I think you're confusing window size and packet size.  In my tests I sent
multimegabyte files with an infinite window size (to get rid of all the Acks
== window adjusts), but used 8K packets built up on the fly.  No need to
buffer any more than you want to.

The transmission model I'm proposing is exactly what's used in Zmodem (since
I'm using old modem-transfer protocols as analogies), whose flow control is
"Keep sending data until I cry uncle".  In other words when everything is
going fine (as it is most of the time), you transfer data at the maximum rate
the link can take.  If there's a problem, you can stop on a dime.  This gives
you maximum transfer rate *and* the same control as minimum-size windows with
vast numbers of Acks sent.

>I suspect that sending a block-at-a-time can give good performance if you
>choose a good block size and pipeline transmissions, and if problems with the
>underlying transport are addressed.

This has historically been *very* hard to get right.  Look at the infinite
number of enhanced-flow-control protocols that followed after the X/Y/Zmodems,
all with their own weird quirks and all of which were extremely difficult to
write compatible (and high-performance) implementations for because they
required extensive tuning to get them going just right, and no two
implementors could agree on how to do it, which resulted in either poor
performance or non-interoperability.  You can certainly think up ways to make
it go well in theory, but long experience has shown that it's extremely
difficult to reduce into practice.  OTOH the "Keep sending data until I cry
uncle" model is pretty trivial to get right (I'm actually tempted to hack
OpenSSH for this just to get the 4-5x faster transmission to one server I use
in the US - sorry guys :-), and the model has been heavily tested in Zmodem
over many years and found to be very workable (I hacked it together in no time
at all for an SFTP implementation that could politely be described as
"embarrasingly crude").

Peter.