RE: Text file type hint proposal for filexfer

["denis bider" <ietf-ssh%denisbider.com@localhost>]

Hi Jacob, all,

I think you are right on all counts.

I agree that attrib flags for text files are OK as proposed.

About server-side heuristics, I agree they would need to be optional, and if
they're optional the client cannot rely on their presence, so they wouldn't be
very useful in a general case. Also, a client supporting auto-mode transfers
would need to implement heuristics and conversion internally anyway for
interoperability with currently existing servers.

(Note - when I say a client supporting auto-mode transfers, I do not mean a
client not giving the user a choice. Auto mode is usually accompanied with text
and binary choices. However, I believe that a user will mostly prefer the
program to make the right decision for him if it can, and will choose auto-mode
most of the time.)

If the attrib flags go through though - and I hope they do - I would like to
propose this addition:

  Files not advertised by the server as textual in the
  attrib flags MUST NOT be opened with the SSH_FXF_TEXT flag.
  A server MAY fail textual open requests for files which
  were not advertised as textual in attrib-flags.

This simplifies the server significantly, and does not affect the client.

The assumption for this proposal is that on platforms where file type
(text/bin) is not determined (Unix, Windows), line endings are LF/CR or a
combination (CRLF, LFCR), and the client can convert from any of these formats
into its native format with a simple filter (interpret the first LF/CR as line
end, discard complementary CR/LF if follows - this algorithm could be mentioned
as a good solution in the spec BTW).

My above proposal is ill-conceived only if there is a platform where:
* file type (text/bin) is not determined, and
* line endings are not one of LF/CR or a combination (CRLF, LFCR).

If anyone is aware of such a platform, please let me know. If there isn't one
though, I suggest the server shouldn't have to support SSH_FXF_TEXT if file
type isn't determined. The client can read binary and convert line endings if
it so pleases.

denis


> -----Original Message-----
> From: ietf-ssh-owner%NetBSD.org@localhost 
> [mailto:ietf-ssh-owner%NetBSD.org@localhost] On Behalf Of Jacob Nevins
> Sent: Monday, October 11, 2004 00:56
> To: ietf-ssh%netbsd.org@localhost
> Subject: Re: Text file type hint proposal for filexfer
> 
> 
> denis bider <ietf-ssh%denisbider.com@localhost> writes:
> > I would like to note that checking whether the file is 
> textual or not
> > is a cheap operation only on VMS.
> > 
> > Defining the text/binary hint as a file attribute makes it 
> less likely
> > that Unix and Windows servers will implement this.
> 
> My intention in defining the attribute extension was to support those
> servers which have a reasonably authoritative text/binary distinction,
> for example in the filesystem, rather than those which have to use a
> heuristic approach, such as Unix/Windows, where I would much 
> prefer that
> UNKNOWN is returned rather than a guess made.
> 
> > I think we would benefit if it were possible for a client 
> to query the
> > likely textual or binary state of a file independently of anything
> > else. In other words, without obliging the server to either 
> show this
> > possibly expensive information as part of regular easily-obtainable
> > attrib-bits, or not support this feature at all.
> > 
> > I think the attribute hints are a nice idea and will work for VMS.
> > However for Unix and Windows servers, we might also benefit 
> from a way
> > for the client to prompt the server to determine on the fly whether
> > the file is textual or binary, if the server has not indicated the
> > file type in attrib-bits. This query would be done right before
> > downloading the file.
> 
> I'd prefer any such extension to be optional, and to have defined
> semantics of being heuristic rather than authoritative if that's how
> it's implemented; as a client (or rather client user), I would like to
> be able to distinguish heuristic from authoritative 
> information so that
> I can choose not to use the former while still getting the benefit of
> the latter. (Perhaps the extension could have a flag 
> indicating whether
> it contains "authoritative" information?)
> 
> I should come out and say that my personal view is that I'd rather not
> trust the integrity of my files to text/binary distinction heuristics,
> which is why I'm pushing for this distinction. I realise this isn't a
> universal view, but I don't think I'm alone in this; it's a popular
> feature but I'd rather it was possible for the end user to choose
> whether to use it.
> 
> > However if there was widespread server-side support for a query to
> > determine file type, the detection and conversion 
> algorithms could be
> > entirely removed from the client.
> 
> Except that the client would need to convert from canonical form.
> I agree that if you're going to do file type detection on 
> server->client
> transfers, it's better to do it on the server.
> 
> (Clients will also need to have some means of file type detection for
> client->server transfers if they don't have that information to hand.)
> 
> > My thinking is, either we require Unix and Windows servers to
> > implement both (a) conversion - SSH_FXF_TEXT flag, AND (b) file type
> > detection; or we should require them to do neither.
>   [...]
> > The idea is, let's be consistent, and let's not specify 
> things halfway
> > through.
> 
> I don't think there's anything inconsistent here.
> 
> The server may not know _which_ files are text files, but it 
> could well
> be in a position to know what the likely line ending convention is if
> the user asserts that given file _is_ a text file, which is a better
> position than the client is in; in such a situation the server should
> convert to canonical form iff requested. This is the position I see
> simple (non-heuristic) Unix/Windows servers as being in.
> 
> Given the which, having no text-hint-attrib-bits support and 
> yet having
> SSH_FXF_TEXT support does make sense; it leaves you in the similar
> situation to plain FTP (or SFTP before this extension) of the user
> having to decide which are text files, which is IMO where you 
> should be
> (or at least have the option of being) with such servers.
> 
> That said, I wouldn't object to making SSH_FXF_TEXT support 
> optional in
> general.
>