Re: Text file type hint proposal for filexfer

[Joseph Galbraith <galb-list%vandyke.com@localhost>]

How about this alternative-- sftpv5 has (and naturally v6 will have)
an 'attrib-bits' field in the attrib structure which describes
various bit attributes, such as case-sensitivity, if the file
is encrypted on disk, compressed on disk, advisory readonly,
a hidden file, a system file, etc.

What if we add two bits there:

    #define SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT      0x00000800

       The server is reasonably certain that the file
       contains textual data, and therefore the file
       should be opened in text mode.

       This flag MUST NOT be present during a setstat operation.
       If this flag is present during an fsetstat operation,
       the file handle is converted to a text-mode handle, as
       if it had been opened with SSH_FXF_ACCESS_TEXT_MODE.

       The server MUST NOT set this bit unless it is reasonably
       certain the file should be transfered in text-mode because
       many clients will use this flag to initiate automatic
       text-mode translation.  If the server sets this flag
       in error, data corruption will result.

    #define SSH_FILEXFER_ATTR_FLAGS_BIN_HINT       0x00001000
       The server is reasonably certain that the file
       contents are binary and should not be opened
       with SSH_FXF_ACCESS_TEXT_MODE.

This gets around your atomicity issue (I think.)  The client
can open sans SSH_FXF_ACCESS_TEXT_MODE, do an fstat, and then,
if the server comes back with SSH_FILEXFER_ATTR_FLAGS_TEXT_HINT
in the attrib.attrib-bits, it can do a fsetstat to put the
handle into text mode.

mime types have come up enough times that I think I'm
going to add a field for them; anybody that doesn't
want shouldn't need to implement them though.

(I know of at least one filesystem that does know the
mime-type information though.)

- Joseph

Jacob Nevins wrote:
> SFTP v4 (draft-ietf-secsh-filexfer-04) added the FXF_TEXT flag to OPEN
> to allow the client to request that the server open a file in text
> mode.
>
> However, no means is provided for the server to advise the client on
> whether this transfer mode is appropriate. In all cases it is likely
> up to the user to manually specify which mode to use.
>
> In FTP, this has tended to lead to corrupted file transfers when
> unintentional translation took place, etc. Some FTP implementations
> have performed "automatic" conversions, e.g., using heuristics based
> on file contents; this is not reliable, and anyway rather defeats the
> point of having the server translate to a known representation.
>
> The following proposal allows the server to optionally communicate
> this information to the client via attributes. (Since it's an
> attribute, the client can also manipulate it on the server as far as
> is specified here if it corresponds to state there.)
>
> (This proposal is against draft-ietf-secsh-filexfer-05.)
>
> To section 5 "File Attributes", 2nd para, add between `attrib-bits' and
> `extended_count':
>
>        byte     content_type         present only if flag CONTENT_TYPE
>
> To section 5.1 "Flags", add:
>
>        #define SSH_FILEXFER_ATTR_CONTENT_TYPE      0x00000400
>
> Add new section after section 5.8 "attrib-bits":
>
> 5.x Content type
>
>    The `content_type' field, if present, indicates whether the file is
>    known to be a text file, known _not_ to be a text file, or is of
>    unknown content.  When sent from server to client, it acts as a
>    hint to the client as to whether a file should be opened in
>    SSH_FXF_TEXT mode.  The following values are defined:
>
>         #define SSH_FILEXFER_CTYPE_UNKNOWN         0
>         #define SSH_FILEXFER_CTYPE_BINARY          1
>         #define SSH_FILEXFER_CTYPE_TEXT            2
>
> To the end of the description of SSH_FXF_TEXT in section 6.3.1
> "Opening a File", add:
>
>       Clients MAY decide whether to use SSH_FXF_TEXT based on a
>       previously seen `content_type' attribute; see Section 5.x.
>
> (Of course, this could all be easily reformulated as an extension
> attribute if desired.)
>
> A version of filexfer-05 marked up with this proposal can be found at
> <http://www.chiark.greenend.org.uk/ucgi/~jacobn/cvsweb/ssh-filexfer-filetype/draft-ietf-secsh-filexfer-05-plus-filetype.txt.diff?r1=1.3&r2=1.4&f=H>
> or <http://tinyurl.com/4cbfz>.
>
> Warts with this proposal:
>
> It's unfortunate that this can't be made atomic within the existing
> protocol design; it requires participating clients to do a STAT or
> similar before OPEN.
> (In fact, perhaps there should be some guidance on which of STAT or
> LSTAT to do in this case.)
> I don't know whether the non-atomicity would be a problem in practice.
>
> It's perhaps tempting to use the IETF's usual content-type notation,
> "MIME types" (RFCs 2045-9 and related standards). However:
>  - I don't know of any real filesystems which use it.
>  - Consider "multipart/mixed" and friends.
>  - Does everything in "text/*" want to be opened FXF_TEXT?
>  - Does everything outside "text/*" _not_ want to be opened FXF_TEXT?
>    (Excluding composite types like multipart.)
>  - It risks adding further to SFTP's bloat if not specified carefully.
>    We don't want to end up accidentally requiring implementations to
>    contain entire MIME implementations (or more likely, ill-defined
>    subsets thereof with poor interoperability).