Re: Why SFTP performance sucks, and how to fix it

['Martin Pool' <mbp%samba.org@localhost>]

On  8 Jul 2003, Howard Chu <hyc%highlandsun.com@localhost> wrote:
> > From: ietf-ssh-owner%netbsd.org@localhost [mailto:ietf-ssh-owner%netbsd.org@localhost]On Behalf
> Of Martin Pool
> 
> > On  8 Jul 2003, Peter Gutmann <pgut001%cs.auckland.ac.nz@localhost> wrote:
> > > Another thing to keep in mind here is that if you signal a
> > > read/write of the
> > > entire file at once, the receiver knows how much disk space
> > > it needs and can
> > > lock the space before starting the receive
> >
> > I don't know of a means for a Unix or Windows server to "lock" disk
> > space in that way.  I suppose the server might zero-fill the blocks
> > between getting the start of the request and reading the bulk of it,
> > but that seems highly contrived.
> 
> The FTP protocol has an ALLO command for allocating space in advance. On some
> FTP servers I've seen, they do exactly this - zero-fill that much space. It
> is a pretty foolproof way of accomplishing the task, even if it seems
> contrived.

There are realistic cases where it might fail: compressed filesystems
might be able to store 2G of zeros, but not 2G of real data; versioned
filesystems may run out of space when rewriting an existing file; or
the user might be in a grace period for disk quotas that expires
during the transfer.  It would give some assurance that there is
enough space, but it's not foolproof.

But my main point is that it is *unavoidable* that disk operations can
fail during writing.  Out of space is only one possibility; hardware
errors are another.  

A protocol that relies on writing the whole file in one operation can
only handle this by either wasting time and bandwidth on an operation
that is known to have failed, or by dropping the connection without
(inline) explanation.  

Another advantage of course is that the client can reconnect and
resume a transfer.

> > > I'd really prefer to know before I start that a transfer is going to
> > > fail, rather than write 2GB and then get the out-of-disk error.
> 
> Definitely.

I suppose you could usefully add a preallocation operation as an
extension command, but it won't be a guarantee.

-- 
Martin