fd forwarding, take 2

[der Mouse <mouse%Rodents-Montreal.ORG@localhost>]

After mulling over the conversation, I decided there were basically two
approaches: (1) use a separate channel for each data stream, and (2)
keep multiple data streams within a single channel.  (1) gets the flow
control for free, but imposes complexity in keeping multiple channels
together; (2) gets the converse: keeping the data streams together
comes for free, but requires inventing a way to flow-control distinct
streams separately within a single channel.

I decided I prefer (2); it looks simpler to me to provide split-apart
flow control than to link multiple channels together.  Based on that,
I've written up a new version of the document I floated before.  I also
included a per-data-stream version of OpenSSH's eow%openssh.com@localhost.  (The
description of fd-forward@ requests is basically unchanged.)

As before, comments are invited.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse%rodents-montreal.org@localhost
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

This file describes some private requests and the facilities they are
designed to provide.  Specifically, it documents
fd-forward%rodents.montreal.qc.ca@localhost, data-eof%rodents.montreal.qc.ca@localhost,
data-eow%rodents.montreal.qc.ca@localhost, and
split-window%rodents.montreal.qc.ca@localhost.

fd-forward%rodents.montreal.qc.ca@localhost is both a global request and a
channel request.  data-eof%rodents.montreal.qc.ca@localhost is a channel request.
data-eow%rodents.montreal.qc.ca@localhost is a channel request.
split-window%rodents.montreal.qc.ca@localhost is both a global request and a
channel request.  While they are all related, in that they synergize
with one another, they are, strictly, orthogonal, so they are described
separately.

For brevity, the text below omits the @rodents.montreal.qc.ca part of
these requests, speaking (for example) of an fd-forward request.

There are various MOUSSH_* constants used in the descriptions below.
Their values are given in a table at the end of this document.

---------------- fd-forward%rodents.montreal.qc.ca@localhost

Standard ssh has no way to handle running remote processes with more
than one input channel (stdin, implemented with SSH_MSG_CHANNEL_DATA
from client to server) or more than two output channels (stdout,
implemented with SSH_MSG_CHANNEL_DATA from server to client, and
stderr, implemented with SSH_MSG_CHANNEL_EXTENDED_DATA of type
SSH_EXTENDED_DATA_STDERR from server to client).  This request supports
forwarding other file descriptors in either or both directions.  It is
fundamentally based on a file descriptor model more or less like
POSIX's; non-POSIX OSes will find it difficult to implement to
approximately the degree that they do not match the POSIX file
descriptor model.

As a global request, fd-forward may be sent only by the client.  It is
used to enquire whether the server supports this protocol; this allows
the client to detect servers that don't support the protocol earlier
than the use of the channel request does.  (If the client doesn't care
about that, it can skip issuing the global request and just error out
if the channel request is rejected.)  In this use, it MUST have
want-reply set true and MUST have zero bytes of request-specific data;
servers supporting the protocol described here MUST respond to such a
request with with SSH_MSG_REQUEST_SUCCESS and zero bytes of
response-specific data.  Global requests with want-reply false and/or
nonempty request-specific data are reserved for future specification;
servers MUST respond to any such with SSH_MSG_REQUEST_FAILURE.  Clients
MUST respond to any fd-forward global requests with
SSH_MSG_REQUEST_FAILURE, regardless of want-reply and associated data.

As a channel request, fd-forward may be issued by the client on any
channel of type "session" which has not yet had an "exec" or "shell"
request (or any other requests of similar semantics - collectively,
exec-style requests) made on it.  (moussh implements no exec-style
requests other than standard exec and shell requests as of this
writing.)  It also MUST have want-reply set true.  Using it in
violation of any of these requirements (on a channel of any other type,
on a "session" channel which has had an exec-style request made on it,
or with want-reply false) is reserved for future specification and MUST
draw SSH_MSG_REQUEST_FAILURE.  The server may also issue fd-forward
channel requests, but only as outlined below, in response to
client-issued fd-forward channel requests; other use in the
server->client direction is similarly reserved and likewise MUST elicit
SSH_MSG_REQUEST_FAILURE.  The same is true of requests in either
direction which meet the above criteria but carry request-specific data
(or lack thereof) violating the syntax below.

Data for forwarded fds is carried in SSH_MSG_CHANNEL_EXTENDED_DATA
messages.  The data_type_code values for these messages are selected,
at forwarding setup time, by the sender of the data.  Current
expectation is that they will be selected from the private-use range
0xfe000000-0xffffffff, but this is not a requirement of this
specification; if suitable IETF standardization occurs, they may be
drawn from the 0-0xfdffffff range.  (A descriptor which carries data in
both directions thus has two associated data_type_codes, one for each
direction.)

Forwarding of multiple fds may be requested by sending multiple channel
requests or by sending multiple blobs in a single channel request, or
any combination of the two.  However, requesting forwarding more than
once for a given fd, whether in a single message or in different
messages, is an error.  In this case, the server is permitted to reject
all of the relevant requests; it is also permitted to accept one of
them and reject the rest.  Similarly, requesting multiple forwardings
using identical data_type_code values is an error; at most one of any
such set may be accepted.  In either case, if one is accepted, there is
no requirement which one it is; it will usually be the first, for some
sense of "first" convenient for the implementation, but this is not a
requirement - the only specification here is that at most one request
be accepted for a given fd on a given channel, and at most one for any
given data_type_code on a given channel.  (This applies even if the
requests in question specify identical forwardings.)

The request-specific data in the request MUST consist of

	byte	MOUSSH_FDFWD_MSG_REQUEST

followed by a stream of forwarding request blobs.  Each of these blobs
consists of

	uint32	fd
	byte	flags
	(additional data may follow; see below)

fd is the descriptor to be forwarded.  (This protocol has no support
for descriptor numbers that do not fit in a uint32.)  flags is a
bitmask of flags:

	0x01	Input
	0x02	Output
	0x04	Inessential
	0xf8	Reserved for future specification, must be 0

If the input flag is set, data may flow from client to server.  In this
case, there is one uint32 of additional data after the flags byte,
which is the data_type_code for data on this fd in this direction.  If
the input flag is clear, there is no way for data to flow in the
client-to-server direction on this fd.

If the output flag is set, data may flow from server to client.  The
data_type_code for this data is selected by the server and is returned
in the response (see below).  If the output flag is clear, there is no
way for data to flow in the server-to-client direction on this fd.

If the inessential flag is set, failure to establish this forwarding
may be benign; if clear, it is always fatal.  See below for details.

It is legal (though mostly useless) for a request to carry zero request
blobs, ie, for its request-specific data to contain just the
MOUSSH_FDFWD_MSG_REQUEST byte.  Such a request must be responded to as
usual, with the response carrying zero response blobs.

It is legal for a request blob to specify any of the four possible
combinations of the input and output flags, but, depending on the
facilities available on the server, some of them may cause the
forwarding request to be rejected.  (Given only the facilities
specified here, a forwarding with neither input nor output enabled is
fairly useless, but there's no reason to forbid it, and someone may
find a use for it, or future expansion of this protocol may make such
forwardings useful.)

Because these requests MUST have want-reply set true, they will always
draw an ssh-level response.  If the request conforms to the above
syntax, this MUST be SSH_MSG_CHANNEL_SUCCESS, and the server MUST
generate a channel request in the other direction as described below.
Otherwise, the ssh-level response MUST be SSH_MSG_CHANNEL_FAILURE, and
the server generates no further response and does not process any of
the forwarding request blobs contained in the erroneous message (in
particular, it MUST back out any changes made due to processing
non-erroneous blobs before noticing the error).

The channel request generated by the server in response to a
MOUSSH_FDFWD_MSG_REQUEST request which draws SSH_MSG_CHANNEL_SUCCESS
MUST be sent with want-reply false; its request-specific data contains

	byte	MOUSSH_FDFWD_MSG_RESPONSE

followed by a stream of forwarding response blobs, exactly one for each
request blob in the request; they are in corresponding order (that is,
if request blob X occurs before request blob Y in the
MOUSSH_FDFWD_MSG_REQUEST data, then the response blob for request blob
X must occur before the response blob for request blob Y in the
resulting MOUSSH_FDFWD_MSG_RESPONSE).  MOUSSH_FDFWD_MSG_RESPONSE
messages MUST always occur in the same order as the
MOUSSH_FDFWD_MSG_REQUEST messages they are responses to.  This
MOUSSH_FDFWD_MSG_RESPONSE fd-forward channel request MUST occur after
the SSH_MSG_CHANNEL_SUCCESS generated for the corresponding
MOUSSH_FDFWD_MSG_REQUEST fd-forward channel request.

Each response blob contains either a failure indication, with a reason
message, or a success indication, possibly with a data_type_code
(depending on whether the corresponding request blob had its output bit
set).

If the forwarding request is rejected, the response blob contains

	byte	MOUSSH_FDFWD_REQ_REJECTED
	string	reason

The reason string gives the reason for the rejection, if the server
chooses to provide one; if not, it is zero-length string.  (RFC4251
section 4.5 is relevant to the generation of these strings.)  If the
corresponding request blob's inessential bit was set, this failure may
(but not must) be benign, in which case the server should operate as if
the corresponding forwarding request had not been made.  If the
inessential bit was clear, or it was set but the server chooses to
consider the failure fatal nevertheless, then the server MUST reject
any following exec-style request on this channel.

If the forwarding request is accepted and the corresponding request
blob's output flag bit was clear, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED

If the forwarding request is accepted and the corresponding request
blob's output flag bit was set, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED
	uint32	data_type_code

where the data_type_code will be used for server-to-client data for
this fd.

There are often some conditions which can cause a forwarding to fail
but which cannot be detected until an exec-style request is
attempted.  To deal with these cases, if any fd forwarding requests
were accepted on a channel, then an exec-style request on that channel
MUST cause the server to generate, before its normal response to the
request, one or more fd-forward channel requests, with want-reply set
false, whose request-specific data consists of

	byte	MOUSSH_FDFWD_MSG_STATUS

followed by a stream of zero or more status blobs.  The server MAY
generate MOUSSH_FDFWD_MSG_STATUS requests bearing zero status blobs,
even if there have been no accepted forwardings (though they are also
permitted even if there have been accepted forwardings).  If the client
has not sent any fd-forward requests at all, status requests must be
zero-blob (see below), and the server MUST be prepared for them to
fail, because the client may not implement this protocol at all.
Clients MUST NOT fail zero-blob status messages, even if they have not
requested any forwardings.  (There's no real use for such
MOUSSH_FDFWD_MSG_STATUS requests from a protocol standpoint, but
implementations may find them convenient.)

Each status blob contains the status for one fd for which a forwarding
request has been accepted; each accepted forwarding must be represented
exactly once in the status blobs sent, though there are no restrictions
on the order in which the status blobs occur, and no restrictions
except message size on the way the collection of status blobs are
grouped into MOUSSH_FDFWD_MSG_STATUS messages.  (This is why only
zero-blob staus messages are permitted if no forwardings have been
accepted: there are no forwarding requests to put the status of in any
status blobs.)  However, all accepted forwarding requests' status blobs
MUST be sent before the usual response to the exec-style request
occurs, and MOUSSH_FDFWD_MSG_STATUS requests MUST NOT be sent under any
other circumstances (before receiving an exec-style request or after
sending the usual response to it, or sent by the client under any
circumstances).

The status blob for a forwarding which succeeds contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_WORKED

whereas that for a forwading which fails contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_FAILED
	string	reason

with the reason string being as described for
MOUSSH_FDFWD_REQ_REJECTED.

Note that if a forawrding request whose inessential bit was clear fails
at this point, the exec-family request MUST fail, even if it otherwise
would have succeeded.

It is not a violation of this spec to use these facilities to request
forwarding for fd numbers 0, 1, or 2, but most servers will reject such
attempts (because they use those fds for the standard protocol's input,
output, and error streams).

Question: is it reasonable to require generating an
SSH_MSG_REQUEST_FAILURE in response to a request with want-reply set
false?  Perhaps ths spec should specify less about want-reply?  The
standard's text appears somewhat ambiguous.

---------------- data-eof%rodents.montreal.qc.ca@localhost

Standard ssh has no way to indicate EOF on one data flow without
indicating EOF on the entire channel.  For input, this is not a
problem, since there is only one data flow, but it is an issue for
output, and, if any input fds are forwarded with fd-forward, it can be
a problem for input too.  We deal with this issue with data-eof channel
requests.

To indicate EOF on the main data stream, the one carried via
SSH_MSG_CHANNEL_DATA, a sender sends a data-eof request whose
request-specific body consists of

	byte	MOUSSH_EOF_MAIN

To indicate EOF on an extended data stream, one carried via
SSH_MSG_CHANNEL_EXTENDED_DATA, the sender sends a data-eof request
whose request-specific body consists of

	byte	MOUSSH_EOF_EXTENDED
	uint32	data_type_code

where data_type_code is, of course, the data_type_code whose extended
data stream is being terminated.  It is a protocol error to send an
SSH_MSG_CHANNEL_DATA message on a channel after sending MOUSSH_EOF_MAIN
on that channel, or SSH_MSG_CHANNEL_EXTENDED_DATA with a particular
data_type_code after sending MOUSSH_EOF_EXTENDED with that
data_type_code.

These are valid with either setting of want-reply, though of course
they will normally be sent with want-reply set true.  If the request is
rejected, it is up to the sender to decide what to do.  In some
circumstances it will be appropriate to ignore the failure, allowing
the data stream to remain in operation as far as the receiver is
concerned; in others, a more appropriate action would be to send
SSH_MSG_CHANNEL_EOF, closing down all data streams in that direction.

Like SSH_MSG_CHANNEL_EOF, MOUSSH_EOF_MAIN and MOUSSH_EOF_EXTENDED do
not require or consume channel window space.  Also like
SSH_MSG_CHANNEL_EOF, indicating EOF on a data stream has no effect on
any data flowing in the other direction.

---------------- data-eow%rodents.montreal.qc.ca@localhost

Standard ssh has no way to indicate "don't send me anything more", what
shutdown(2) with SHUT_RD indicates - or, from the perspective of the
sender, the status indicated by EPIPE errors or SIGPIPE signals.  The
only thing a receiver can do under these circumstances is tear down the
channel entirely, which is excessively drastic for many circumstances.

data-eow channel requests fill this gap.  They may be sent at any point
when window adustments are permitted.  The body of a data-eow request
consists of

	byte	MOUSSH_EOW_MAIN

To indicate this status on the main data stream, the one carried via
SSH_MSG_CHANNEL_DATA, or

	byte	MOUSSH_EOW_EXTENDED
	uint32	data_type_code

to indicate this status on the extended data stream corresponding to
SSH_MSG_CHANNEL_EXTENDED_DATA with the same data_type_code.

After sending one of these messages, an implementation MUST discard any
further data received on the stream in question; if split-window (see
below) is in use, then it also MUST NOT send any further window
adjustments for the data stream in question.

If the receiver of such a message rejects it, it is up to the sender to
decide how to handle the condition, though simply ignoring sent data
will probably be the best option in most cases.  (If not using
split-window, it must be consumed to avoid blocking other flows; if
using split-window, it will usually be reasonable to simply let
back-pressure block that data stream indefinitely.)

Loosely speaking, this request is to eow%openssh.com@localhost what data-eof is
to standard SSH_MSG_CHANNEL_EOF.  That is, it is a data-stream-specific
version of functionality available for the whole channel via other
mechanisms.

---------------- split-window%rodents.montreal.qc.ca@localhost

Standard ssh has only one data flow-control window per direction per
channel.  This is sufficient in simple cases, but it raises problems
when multiple fds are forwarded, and in unusual cases it may even cause
trouble between stdout and stderr when not forwarding additional fds.
split-window addresses this.

When split-window is in use on a channel in a given direction, there
is, in that direction, one flow-control window per data stream, not a
single shared one for the whole channel.  To propose entering this
mode, a data sender sends a split-window channel request with
want-reply true whose body is

	byte	MOUSSH_SPLIT_START

If the peer who receives this packet does not implement split-window,
or does not choose to allow it for this direction on this channel, it
rejects the request; otherwise, it accepts it.  This request MUST NOT
be sent after sending any SSH_MSG_CHANNEL_DATA or
SSH_MSG_CHANNEL_EXTENDED_DATA messages on the channel, or after it has
already been accepted once.  Entering this mode resets all data streams
to have zero window (the initial window value provuded during channel
open is ignored).  Also, after accepting a MOUSSH_SPLIT_START, a data
receiver MUST NOT send SSH_MSG_CHANNEL_WINDOW_ADJUST on that channel.

Peers sending MOUSSH_SPLIT_START must be prepared to receive
traditional window adjustments before receiving the response to the
MOUSSH_SPLIT_START.  If the split request is accepted, any such
adjustments will be discarded; if not, they must of course be preserved
as if the split request had not been made.

To provide additional window, a data receiver sends a split-window
channel request with want-reply false whose body is

	byte	MOUSSH_SPLIT_ADJUST_MAIN
	uint32	bytes to add

or

	byte	MOUSSH_SPLIT_ADJUST_EXTENDED
	uint32	data_type_code
	uint32	bytes to add

according as the window is to be provided for the main data stream (the
one which uses SSH_MSG_CHANNEL_DATA) or a secondary stream (one which
uses SSH_MSG_CHANNEL_EXTENDED_DATA), providing the appropriate
data_type_code for the latter case.

These work just like SSH_MSG_CHANNEL_WINDOW_ADJUST except that they
affect only the relevant stream's window (in particular, the same
4294967295-octet limit applies).  Windows are independent; one data
stream's window has no effect whatever on the legality of sending data
on another, even if they're part of the same channel.

There is no way to take a channel out of split-window mode, short of
destroying it entirely.

Note that split-window mode is enabled per-direction.  Enabling it in
one direction has no effect on its use (or lack thereof) in the other.

---------------- MOUSSH_* values

Here are the various MOUSSH_* values.  Note that these sometimes are
distinct even though they don't need to be; for example,
MOUSSH_FDFWD_MSG_* values occur in a different context than
MOUSSH_FDFWD_REQ_* values and thus could reuse numeric values.  This is
done this way largely to improve ease of finding bugs.  But note that,
for example, the MOUSSH_FDFWD_* values used by fd-forward messages do
overlap with the MOUSSH_EOF_* values used by data-eof messages; this is
because they are conceptually orthogonal capabilities.

	MOUSSH_FDFWD_MSG_REQUEST	1
	MOUSSH_FDFWD_MSG_RESPONSE	2
	MOUSSH_FDFWD_MSG_STATUS		3
	MOUSSH_FDFWD_REQ_ACCEPTED	4
	MOUSSH_FDFWD_REQ_REJECTED	5
	MOUSSH_FDFWD_STAT_WORKED	6
	MOUSSH_FDFWD_STAT_FAILED	7

	MOUSSH_EOF_MAIN			1
	MOUSSH_EOF_EXTENDED		2

	MOUSSH_EOW_MAIN			1
	MOUSSH_EOW_EXTENDED		2

	MOUSSH_SPLIT_START		1
	MOUSSH_SPLIT_ADJUST_MAIN	2
	MOUSSH_SPLIT_ADJUST_EXTENDED	3