Re: Why SFTP performance sucks, and how to fix it

[Nicolas Williams <Nicolas.Williams%sun.com@localhost>]

On Tue, Jul 08, 2003 at 03:38:48PM +1000, Martin Pool wrote:
> On  8 Jul 2003, Peter Gutmann <pgut001%cs.auckland.ac.nz@localhost> wrote:
> > Martin Pool <mbp%samba.org@localhost> writes:
> >
> > Fair enough.  I guess the XON/XOFF thing (to replace the Ack) would do this,
> > or perhaps a channel close depending on the seriousness of the
> > error.
> 
> Channel close is a terrible error handling method.  To start with, it
> gives no indication of what in particular went wrong.

Indeed.  SFTP is not like FTP where there's separate channels for
control data and data - closing the channel means killing the session
with no useful error indication.

> I don't know of a means for a Unix or Windows server to "lock" disk
> space in that way.  I suppose the server might zero-fill the blocks
> between getting the start of the request and reading the bulk of it,
> but that seems highly contrived.

And inefficient.

> I suspect that sending a block-at-a-time can give good performance if
> you choose a good block size and pipeline transmissions, and if
> problems with the underlying transport are addressed.  Perhaps IOs
> should be roughly proportional to the amount of data in flight?  I
> think the request/response model in SECSH is a great thing for
> simplicity and robustness and it shouldn't be lightly cast aside.

The SFTP protocol is, in some ways, more like a remote filesystem
protocol than like a file transfer protocol (my observation - I'm not
saying it should be).  Thus a client can interleave read/write requests
relating to multiple files over the same channel.

But note that the client is not required to wait for SFTP "Acks" before
proceeding with other requests.  SFTP clients are allowed[1] to issue
reads/writes for entire files before handling a single reply to any of
them.  Doing so may not be wise for very large file transfers, as Martin
points out.

The fact is that SFTP does not have an Ack "handbrake" - it has Acks,
but no Ack handbrake.  Implementations may, but not the specification.

[1] Or, rather, such client behaviour is not forbidden by the text of
    the I-D.  Pipelining block-at-a-time requests is not disallowed;
    perhaps it should be encouraged.

Cheers,

Nico
--