> The agr(4) man page hints at the issue.  The problem is that you
> *really* don't want TCP segments from a single connection arriving
> out of order.  While TCP semantics guarantee that things will work,
> it will cause a tremendous performance hit.

Surely that's a (performance) bug in the TCP stacks?  The network has
never promised it won't reorder packets, and indeed when I traceroute I
see packets only a second apart taking different paths in terms of
router addresses (surely going down different wires to different
next-hops is even more reordering-prone than going down different wires
to the same next-hop?).

> In particular, if a sender receives 3 duplicate ACKs in a row, it
> slams its congestion window shut and restarts the whole slow start
> business.

Wouldn't it make more sense to either be less hair-trigger in the
congestion detection algorithms or also implement some kind of
reordering detection to avoid this?  Assuming the network won't reorder
packets really strikes me as a modern version of the "everything is on
a fast LAN" problem that many Berkeley network programs suffered from,
and which got smoked out once IP started getting used over longer
paths.

If none of those are considered usable for some reason, then I guess
the world needs a link aggregation design that preserves packet order
but still load-balances sanely.  In NetBSD kernel terms, I wonder if
all the member interfaces could be made to share the same output queue?
You'd mostly get lack of reordering *and* still have real
load-balancing, it seems to me.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B