Re: Draft draft of coming sftp revision...

["Joseph Galbraith" <galb-list%vandyke.com@localhost>]

> > It seems like it has been a while since we had a ruckus on the
> > mailing list -- so I thought I'd post a draft-draft of the coming
> > version of the sftp draft.
> 
> First things first: I _like_ the human-readable error message field
> in SSH_FXP_STATUS. This is something that institutions I have
> contact with have been complaining bitterly about for ages - one lot
> refused to provide SFTP to their users _at all_ simply because it
> was unable to report errors with the same richness as SCP. Even if
> every other proposed change gets shouted down, KEEP THIS ONE!

That one actually existed in version 3 of
the protocol, before I started editing 
the draft :-)  But I was a big proponent
of it too.

> >    o  Added 'newline' extension.
> >    o  Added SSH_FXF_TEXT file open flag.
> 
> These seem like a half-decent 90% solution (covers Windows, Mac and
> Unix with no difficulty), but I'm concerned about the other 10% :-)
> 
> For a start, if you consider text file issues to be only about
> newline conventions, you thereby ignore other problems such as
> character sets. If a server uses EBCDIC and a client doesn't, is it
> really the expected behaviour of a text-mode file transfer that it
> should yield an EBCDIC file at the client end or an ASCII file on
> the server? I was vaguely under the impression that (say) FTP's idea
> of a text-mode transfer was that character sets would get converted.
> Granted that EBCDIC is a pretty extreme example, but if one system
> uses ISO-8859-1 while another uses UTF-8, or (more subtly) if two
> systems use UTF-8 with different default normalisation forms, this
> might still be an issue.

I am not a FTP guru, but I just talked to our FTP guru,
and he tells me that stock FTP, at least, doesn't do
any charset translation.  There might be an extension fo
it.

I, personally, don't think we want to solve this one.
I also think that if it's important to you, you can
do it without ietf action -- a couple of possibilities
off the top of my head:

    o include a "charset@x.y.z" extension attribute when
      the file is stat'ed and then let the client deal
      with translation.
    o allow the client to send something like
      SSH_FXP_EXTENDED
      "translate-charset@x.y.z"
      string handle      (must be opened using FXF_TEXT)
      string charset

      and then have the server translate.

Personally, I think the more work we can keep out of the
server the better.

> If the wire format for text files use \r\n as a newline, what should
> an implementation do if it sees a lone \r or a lone \n? It's always
> annoyed me that so many protocol specifications use \r\n as the
> newline and don't mention anything about this situation. Can we have
> a definitive specification of this?

Okay; that's fair.  My first inclination is to
say that anything not matching the servers newline
sequence exactly (either the default \r\n, or newline
as specified by the server) SHOULD NOT be translated.
In particular, if the newline convention is \r\n, and
the client see's a lone \r or lone \n, it should simply
write the data through without modification.

What do you think?

> Seeking in text files worries me a little. If each server can choose
> to define the wire text format to be its own newline convention,
> then this means 90% of servers (as above) can effectively treat text
> files as binary and seeking isn't a problem. But what about things
> like VMS? Don't VMS text files use a fixed-length record per line?
> This surely isn't covered in the 90% solution that is SSH_FXF_TEXT.

This one is tricky-- I think the thing we actually
reached semi-consensus about on the list was that
TEXT files would not be seekable.  However, that
disallows resume of partial transfers.

So, I was trying to allow the following scenerios,
which I suspect fall into the 90% catagory, to work:
    1. Resume aborted transfer
        1a. Allow an aborted FTP / SCP / RCP
            transfer to be resumed
    2. Transfer just the tail of a growing text
       file (a log for example.)

But, usually the only thing we have client
side is the file data.  The trouble with the
fpos_t solution is that in 1a, and 2, and
at least for us, sometimes 1 as well, the
fpos_t would not be available because it
was not written to disk (not file data) and
the application has restarted.

I don't think the current proposal is undoable
for servers where translation is necessary--
it is difficult though.  But, it is a problem
that has been solved by FTP servers for a decade
for the REST (restart) command.

The solution as it currently stands actually works
very much like FTP -- with the exception that
90% of the implementations don't have to do
any work :-)

The server must be able to make following calculation
in order to support seek.

Given:
    o the servers stated newline sequence
    o An offset from the beginning of the data
      as it would appear on the wire

The server must be able to calculate
    o the actual offset in the file.

A client must also be able to do a similar mapping:

Given:
    o the servers stated newline sequence
    o A current offset in it's local file

The client must be able to calculate
    o the offset from the beginning of the data
      stream.

As I said, FTP has been tackling this problem for
a long time.

The other thing I would note is that if a server
want's to support resume, it only needs to support
the offset field on the first write or read.  And
it doesn't need to support that if it doesn't
want to support resume.  I wrote the text to
require clients to expect failure during
non-sequential operation.

Thanks for your feedback, by the way.  I expected
far more feedback than I've gotten -- so it's
nice to know it actually made it to the list :-)

- Joseph