Re: fds beyond 0/1/2

[der Mouse <mouse%Rodents-Montreal.ORG@localhost>]

>> I've got some ideas I can write up and float for comment.

Here's what I have.  Comments are invited.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse%rodents-montreal.org@localhost
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

This file describes the fd-forward%rodents.montreal.qc.ca@localhost and
data-eof%rodents.montreal.qc.ca@localhost requests, and the facilities they are
designed to provide.

fd-forward%rodents.montreal.qc.ca@localhost is both a global request and a
channel request; data-eof%rodents.montreal.qc.ca@localhost is a channel request.
While they are related, in that each is more useful in conjunction with
the other, they are, strictly, orthogonal, so they are described
separately.

For brevity, fd-forward%rodents.montreal.qc.ca@localhost is abbreviated to
fd-forward below; similarly, data-eof%rodents.montreal.qc.ca@localhost is
abbreviated to data-eof.

There are various MOUSSH_* constants used in the descriptions below.
Their values are given in a table at the end of this document.

---------------- fd-forward%rodents.montreal.qc.ca@localhost

Standard ssh has no way to handle running remote processes with more
than one input channel (stdin, implemented with SSH_MSG_CHANNEL_DATA
from client to server) or more than two output channels (stdout,
implemented with SSH_MSG_CHANNEL_DATA from server to client, and
stderr, implemented with SSH_MSG_CHANNEL_EXTENDED_DATA of type
SSH_EXTENDED_DATA_STDERR from server to client).  This request supports
forwarding other file descriptors in either or both directions.  It is
fundamentally based on a file descriptor model more or less like
POSIX's; non-POSIX OSes will find it difficult to implement to
approximately the degree that they do not match the POSIX file
descriptor model.

As a global request, fd-forward may be sent only by the client.  It is
used to enquire whether the server supports this protocol.  In this
use, it MUST have want-reply set true and MUST have zero bytes of
request-specific data; servers supporting the protocol described here
MUST respond to such a request with with SSH_MSG_REQUEST_SUCCESS and
zero bytes of response-specific data.  Global requests with want-reply
false and/or nonempty request-specific data are reserved for future
specification; servers MUST respond to any such with
SSH_MSG_REQUEST_FAILURE.  Clients MUST respond to any fd-forward global
requests with SSH_MSG_REQUEST_FAILURE, regardless of want-reply and
associated data.

As a channel request, fd-forward may be issued by the client on any
channel of type "session" which has not yet had an "exec" or "shell"
request (or any other requests of similar semantics - collectively,
exec-style requests) made on it.  (moussh implements no exec-style
requests other than standard exec and shell requests as of this
writing.)  It also MUST have want-reply set true.  Using it in
violation of any of these requirements (on a channel of any other type,
on a "session" channel which has had an exec-style request made on it,
or with want-reply false) is reserved for future specification and MUST
draw SSH_MSG_REQUEST_FAILURE.  The server may also issue fd-forward
channel requests, but only as outlined below, in response to
client-issued fd-forward channel requests; other use in the
server->client direction is similarly reserved and likewise MUST elicit
SSH_MSG_REQUEST_FAILURE.  The same is true of requests in either
direction which meet the above criteria but carry request-specific data
(or lack thereof) violating the syntax below.

Data for forwarded fds is carried in SSH_MSG_CHANNEL_EXTENDED_DATA
messages.  The data_type_code values for these messages are selected,
at forwarding setup time, by the sender of the data.  Current
expectation is that they will be selected from the private-use range
0xfe000000-0xffffffff, but this is not a requirement of this
specification; if suitable IETF standardization occurs, they may be
drawn from the 0-0xfdffffff range.  (A descriptor which carries data in
both directions thus has two associated data_type_codes, one for each
direction.)

Forwarding of multiple fds may be requested by sending multiple channel
requests or by sending multiple blobs in a single channel request, or
any combination of the two.  However, requesting forwarding more than
once for a given fd, whether in a single message or in different
messages, is an error.  In this case, the server is permitted to reject
all of the relevant requests; it is also permitted to accept one of
them and reject the rest.  Similarly, requesting multiple forwardings
using identical data_type_code values is an error; at most one of any
such set may be accepted.  In either case, if one is accepted, there is
no requirement which one it is; it will usually be the first, for some
sense of "first" convenient for the implementation, but this is not a
requirement - the only specification here is that at most one request
be accepted for a given fd on a given channel, and at most one for any
given data_type_code on a given channel.  (This applies even if the
requests in question specify identical forwardings.)

The request-specific data in the request MUST consist of

	byte	MOUSSH_FDFWD_MSG_REQUEST

followed by a stream of forwarding request blobs.  Each of these blobs
consists of

	uint32	fd
	byte	flags
	(additional data may follow; see below)

fd is the descriptor to be forwarded.  (This protocol has no support
for descriptor numbers that do not fit in a uint32.)  flags is a
bitmask of flags:

	0x01	Input
	0x02	Output
	0x04	Inessential
	0xf8	Reserved for future specification, must be 0

If the input flag is set, data may flow from client to server.  In this
case, there is one uint32 of additional data after the flags byte,
which is the data_type_code for data on this fd in this direction.  If
the input flag is clear, there is no way for data to flow in the
client-to-server direction on this fd.

If the output flag is set, data may flow from server to client.  The
data_type_code for this data is selected by the server and is returned
in the response (see below).  If the output flag is clear, there is no
way for data to flow in the server-to-client direction on this fd.

If the inessential flag is set, failure to establish this forwarding
may be benign; if clear, it is always fatal.  See below for details.

It is legal (though mostly useless) for a request to carry zero request
blobs, ie, for its request-specific data to contain just the
MOUSSH_FDFWD_MSG_REQUEST byte.  Such a request must be responded to as
usual, with the response carrying zero response blobs.

It is legal for a request blob to specify any of the four possible
combinations of the input and output flags, but, depending on the
facilities available on the server, some of them may cause the
forwarding request to be rejected.  (Given only the facilities
specified here, a forwarding with neither input nor output enabled is
fairly useless, but there's no reason to forbid it, and someone may
find a use for it, or future expansion of this protocol may make such
forwardings useful.)

Because these requests MUST have want-reply set true, they will always
draw an ssh-level response.  If the request conforms to the above
syntax, this MUST be SSH_MSG_CHANNEL_SUCCESS, and the server MUST
generate a channel request in the other direction as described below.
Otherwise, the ssh-level response MUST be SSH_MSG_CHANNEL_FAILURE, and
the server generates no further response and does not process any of
the forwarding request blobs contained in the erroneous message.

The channel request generated by the server in response to a
MOUSSH_FDFWD_MSG_REQUEST request which draws SSH_MSG_CHANNEL_SUCCESS
MUST be sent with want-reply false; its request-specific data contains

	byte	MOUSSH_FDFWD_MSG_RESPONSE

followed by a stream of forwarding response blobs, exactly one for each
request blob in the request; they are in corresponding order (that is,
if request blob X occurs before request blob Y in the
MOUSSH_FDFWD_MSG_REQUEST data, then the response blob for request blob
X must occur before the response blob for request blob Y in the
resulting MOUSSH_FDFWD_MSG_RESPONSE).  MOUSSH_FDFWD_MSG_RESPONSE
messages MUST always occur in the same order as the
MOUSSH_FDFWD_MSG_REQUEST messages they are responses to.  This
MOUSSH_FDFWD_MSG_RESPONSE fd-forward channel request MUST occur after
the SSH_MSG_CHANNEL_SUCCESS generated for the corresponding
MOUSSH_FDFWD_MSG_REQUEST fd-forward channel request.

Each response blob contains either a failure indication, with a reason
message, or a success indication, possibly with a data_type_code
(depending on whether the corresponding request blob had its output bit
set).

If the forwarding request is rejected, the response blob contains

	byte	MOUSSH_FDFWD_REQ_REJECTED
	string	reason

The reason string gives the reason for the rejection, if the server
chooses to provide one; if not, it is zero-length string.  (RFC4251
section 4.5 is relevant to the generation of these strings.)  If the
corresponding request blob's inessential bit was set, this failure may
(but not must) be benign, in which case the server should operate as if
the corresponding forwarding request had not been made.  If the
inessential bit was clear, or it was set but the server chooses to
consider the failure fatal nevertheless, then the server MUST reject
any following exec-style request on this channel.

If the forwarding request is accepted and the corresponding request
blob's output flag bit was clear, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED

If the forwarding request is accepted and the corresponding request
blob's output flag bit was set, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED
	uint32	data_type_code

where the data_type_code will be used for server-to-client data for
this fd.

There are often some conditions which can cause a forwarding to fail
but which cannot be detected until an exec-style request is
attempted.  To deal with these cases, if any fd forwarding requests
were accepted on a channel, then an exec-style request on that channel
MUST cause the server to generate, before its normal response to the
request, one or more fd-forward channel requests, with want-reply set
false, whose request-specific data consists of

	byte	MOUSSH_FDFWD_MSG_STATUS

followed by a stream of zero or more status blobs.  If no fd
forwardings were accepted, the server MAY generate one or more
MOUSSH_FDFWD_MSG_STATUS requests bearing zero status blobs (though, if
the client has not sent any fd-forward requests, the server MUST be
prepared for any such requests to fail, because the client may not
implement this protocol at all).  Clients MUST NOT fail such channel
requests, even if they have not requested any forwardings.  (There's no
real use for such MOUSSH_FDFWD_MSG_STATUS requests from a protocol
standpoint, but implementations may find them convenient.)

Each status blob contains the status for one fd for which a forwarding
request has been accepted; each accepted forwarding must be represented
exactly once in the status blobs sent, though there are no restrictions
on the order in which the status blobs occur, and no restrictions
except message size on the way the collection of status blobs are
grouped into MOUSSH_FDFWD_MSG_STATUS messages.  However, all accepted
forwarding requests' status blobs MUST be sent before the usual
response to the exec-style request occurs, and MOUSSH_FDFWD_MSG_STATUS
requests MUST NOT be sent under any other circumstances (before
receiving an exec-style request or after sending the usual response to
it, or sent by the client under any circumstances).

The status blob for a forwarding which succeeds contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_WORKED

whereas that for a forwading which fails contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_FAILED
	string	reason

with the reason string being as described for
MOUSSH_FDFWD_REQ_REJECTED.

Note that if a forawrding request whose inessential bit was clear fails
at this point, the exec-family request MUST fail, even if it otherwise
would have succeeded.

It is not a violation of this spec to use these facilities to request
forwarding for fd numbers 0, 1, or 2, but most servers will reject such
attempts (because they use those fds for the standard protocol's input,
output, and error streams).

This facility has one major known problem: there is only one window for
data per direction, shared by all data streams.  We need a way to
support back-pressure on each forwarded fd independently.  Question:
would it be better to do this within a single channel framework, or
would it be better to open new channels and somehow associate them with
this channel?  I'm leaning towards the former, because we have the same
problem already with stdout and stderr (for example, if you run
"ssh foo producer | consumer" and consumer blocks long enough for pipes
to fill up, an error message from producer on stderr will get blocked
behind the stdout data).

Question: is it reasonable to require generating an
SSH_MSG_REQUEST_FAILURE in response to a request with want-reply set
false?  Perhaps ths spec should specify less about want-reply?  The
standard's text appears somewhat ambiguous.

---------------- data-eof%rodents.montreal.qc.ca@localhost

Standard ssh has no way to indicate EOF on one data flow without
indicating EOF on the entire channel.  For input, this is not a
problem, since there is only one data flow, but it is an issue for
output, and, if any input fds are forwarded with fd-forward, it can be
a problem for input too.  We deal with this issue with data-eof channel
requests.

To indicate EOF on the main data stream, the one carried via
SSH_MSG_CHANNEL_DATA, a sender sends a data-eof request whose
request-specific body consists of

	byte	MOUSSH_EOF_MAIN

To indicate EOF on an extended data stream, one carried via
SSH_MSG_CHANNEL_EXTENDED_DATA, the sender sends a data-eof request
whose request-specific body consists of

	byte	MOUSSH_EOF_EXTENDED
	uint32	data_type_code

where data_type_code is, of course, the data_type_code whose extended
data stream is being terminated.  It is a protocol error to send an
SSH_MSG_CHANNEL_DATA message on a channel after sending MOUSSH_EOF_MAIN
on that channel, or SSH_MSG_CHANNEL_EXTENDED_DATA with a particular
data_type_code after sending MOUSSH_EOF_EXTENDED with that
data_type_code.

These are valid with either setting of want-reply, though of course
they will normally be sent with want-reply set true.  If the request is
rejected, it is up to the sender to decide what to do.  In some
circumstances it will be appropriate to ignore the failure, allowing
the data stream to remain in operation as far as the receiver is
concerned; in others, a more appropriate action would be to send
SSH_MSG_CHANNEL_EOF, closing down all data streams in that direction.

Like SSH_MSG_CHANNEL_EOF, MOUSSH_EOF_MAIN and MOUSSH_EOF_EXTENDED do
not require or consume channel window space.  Also like
SSH_MSG_CHANNEL_EOF, indicating EOF on a data stream has no effect on
any data flowing in the other direction.

---------------- MOUSSH_* values

Here are the various MOUSSH_* values.  Note that these sometimes are
distinct even though they don't need to be; for example,
MOUSSH_FDFWD_MSG_* values occur in a different context than
MOUSSH_FDFWD_REQ_* values and thus could reuse numeric values.  This is
done this way largely to improve ease of finding bugs.  But note that
the MOUSSH_FDFWD_* values used by fd-forward messages do overlap with
the MOUSSH_EOF_* values used by data-eof messages; this is because they
are conceptually orthogonal capabilities.

	MOUSSH_FDFWD_MSG_REQUEST	1
	MOUSSH_FDFWD_MSG_RESPONSE	2
	MOUSSH_FDFWD_MSG_STATUS		3
	MOUSSH_FDFWD_REQ_ACCEPTED	4
	MOUSSH_FDFWD_REQ_REJECTED	5
	MOUSSH_FDFWD_STAT_WORKED	6
	MOUSSH_FDFWD_STAT_FAILED	7

	MOUSSH_EOF_MAIN			1
	MOUSSH_EOF_EXTENDED		2