Re: scp troughput

[Greg Troxel <gdt%lexort.com@localhost>]

I just looked at traces of the TCP connections for ssh and a blaster to
discard, from a fast-cpu sender to an apu2 (a bit slow it turns out) and
an old laptop.  All of this is wired Gbe with decent switches (netgear,
ubiquiti, 1 each).  My "fast" computer is a 2014 dell with re0, so sort
of good stuff but not current and not necessarily the best interface
chip (but perhaps that's not fair).

My tentative conclusions are:

  particularly on the apu2, the tcp window (determined by the
  recvbuf/sendbuf) really matters in terms of 131072 vs 32768

  on the apu2 with discard, the time it takes for the receiver process
  getting the data and updating the window matter a lot.  Even with
  inetd/discard, the window is kept full and the limiting factor seems
  to be userland reading the data.

  with the apu2 or 2008 laptop and 131072 buffers, the limiting issue is
  the sender.  It does not appear to be window constrained.

  with the 2008 laptop and blaster/discard, it's really not window
  limited, and runs at 550 Mb/s.


all rates are in MB/s, with 131072 buffers

ssh:              20.3 apu2, 18.8 2008, 16.1 self
blaster/discard:  31.7 apu2, 69.3 2008, 571.6 self

So I think there is something going on with ssh, perhaps in using kernel
crypto engine.

This is all done with a local build of netbsd-8.  I have in sysctl.conf:

net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.inet6.tcp6.sendspace=131072
net.inet6.tcp6.recvspace=131072

net.inet.tcp.recvbuf_auto=0
net.inet.tcp.sendbuf_auto=0
net.inet6.tcp6.recvbuf_auto=0
net.inet6.tcp6.sendbuf_auto=0

In the past I have become suspicious of the auto buf stuff, but I do not
know if that is fair.


I would definitely recommend bumping up your buffer sizes.   But I
think there's more going on.