Re: solving the SFTP text mode issue

["Joseph Galbraith" <galb-list%vandyke.com@localhost>]

> I think it's close.  There are 3 main fundamental formats we must deal
> with:
>
> - Stream LF (i.e., "UNIX" mode)
> - delimited, variable-length records (usually, CR/LF pairs)
> - variable-length records with a fixed-length binary record-size field
>    at the beginning of each record
>
> There are other permutations of text files in VMS, but they're basically
> "variations on a theme", in that they're a derivation of one of the above
> formats, and so what works for the above should work for them as well.
>
> So, this solution would seem to apply to the 2nd format (the first format
> is obviously handled today), but I'm not sure of the 3rd format.  We would
> need to examine this proposal and see what the implications are.

My model of the world in creating this was:

1. Binary files

   Protocol works as specified.

2. Record oriented files -- where record files have the following
   sub-cases:

   a. Fix length records
   b. Variable length records

      Variable length records might be either delimited
      or counted.

      In the delimited case, the delimiter doesn't really
      matter.  It could be \n, \r, \r\n, or , -- it doesn't
      matter.  (Both your stream LF and delimited variable-length
      records fit into this catagory.)

The solution proposed works extremely well in the
fixed length record cases -- in fact, I would say
it is ideal.  It provides, fast, efficient, random
record based access.

In the variable length record case, it isn't so
great in terms of efficiency.  The server would
probably have to do some optimization -- for
example, preread the file and record where the
record breaks are, or, more likely, simply record
where the last record ended and it's number,
so sequential reads work well.

Counted variable length maps pretty well,
since each string in the array containing
the records has its own count.  I.e., on
the wire it looks like:

Byte    SSH_FXP_RECORD
uint32  32                # starting at record 32
uint32  3                 # 3 records included

uint32  45                # Record 32 is 45 bytes
Byte[45]                  # The 45 bytes for record 32

uint32  143               # Record 33 is 143
BYTE[143]                 # The 143 bytes of data for recrd 33

uint32  1951              # Record 34 is 1951 bytes
BYTE[1951]                # The 1951 bytes of data for record 34

Delimited variable length records look to the client
like counted records, so the client would probably
need some other mechanism to recreate the original
file attributes.  I.e., something during stat
that says this file is variable length, \r\n delimited.

In addition, the delimited case has the additional
challange that the server needs to determine what
the delimiter is -- on VMS, it can probably retrieve
that information from the file attributes.

For the general case, we may want to add the
additional flag:

  SSH_FXF_TEXT_MODE

which implies RECORD_ORIENTED, variable length
records, with the servers native newline character
as the delimiter.

- Joseph

PS. Sorry Dan, I already sent this to you once--
I'm having a bad day with the 'reply all' thing.