Re: Weird network performance problem

[Greg Troxel <gdt%lexort.com@localhost>]

Chavdar Ivanov <ci4ic4%gmail.com@localhost> writes:

>>   It looks like you are using vlan support on Y.  Try without also.
>
> That may be something to look at. This is my NVMM host as well, every
> boot I recreate tap[0..5] for use by the NVMM guests (but the tests
> were done without any of them running).
>
> I am not using vlans deliberately - the switch upstairs is a dumb one,
> although the one downstaris is managed and has (unusued at the moment)
> vlan support. The interfaces are created simply with /etc/ifconfig.wm0
> - just 'inet 192.168.0.29 netmask 255.255.255.0 up description "My
> LAN"' and /etc/ifconfig.bridge0 -

I meant that hardware vlan support was enabled.  I really doubt this is
the issue, but it seems easy to try the easy things.

Also, I forgot my other usual advice.

  $ netstat -s > BEFORE
  $ # do iperf
  $ netstat -s > AFTER
  $ diff -u BEFORE AFTER

to find out which counters that you didn't even know existed are
changing  in perhaps interesting ways.

> From the XCP-NG host to the NetBSD laptop:
>
> $ iperf3 -c ymir.lorien.lan
> Connecting to host ymir.lorien.lan, port 5201
> [  4] local 192.168.0.5 port 36036 connected to 192.168.0.29 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec  45.9 MBytes   385 Mbits/sec    0   66.5 KBytes
> [  4]   1.00-2.00   sec  64.2 MBytes   539 Mbits/sec    0    100 KBytes
> [  4]   2.00-3.00   sec  81.3 MBytes   682 Mbits/sec    0    132 KBytes
> [  4]   3.00-4.00   sec  99.4 MBytes   834 Mbits/sec    0    163 KBytes
> [  4]   4.00-5.00   sec   109 MBytes   911 Mbits/sec    0    205 KBytes
> [  4]   5.00-6.00   sec   111 MBytes   928 Mbits/sec    0    205 KBytes
> [  4]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    0    205 KBytes
> [  4]   7.00-8.00   sec   111 MBytes   932 Mbits/sec    0    205 KBytes
> [  4]   8.00-9.00   sec   111 MBytes   930 Mbits/sec    0    205 KBytes
> [  4]   9.00-10.00  sec   111 MBytes   932 Mbits/sec    0    205 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec   954 MBytes   800 Mbits/sec    0             sender
> [  4]   0.00-10.00  sec   953 MBytes   800 Mbits/sec                  receiver
>
> Starts a bit slower, but after the fourth interval reaches along the maximum.

That's just 1s periods within the same TCP connection.  That is more
expected than subsequent TCP connections.  But, this is a clue of
something to perhaps change.  NetBSD has defaults for send and receive
buffer sizes, and some notion of auto tuning.  I am not really clear
where iperf3 is getting those "Cwnd" values, but perhaps it is able to
obtain them from the Linux kernel?

On some machines I have

net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.inet6.tcp6.sendspace=131072
net.inet6.tcp6.recvspace=131072

net.inet.tcp.recvbuf_auto=0
net.inet.tcp.sendbuf_auto=0
net.inet6.tcp6.recvbuf_auto=0
net.inet6.tcp6.sendbuf_auto=0

which may not be the right thing, but at one point I had trouble with
the auto stuff not rampning up fast enough.

> When the server is the B laptop running W10, I get:
>
> $ iperf3 -c brutus.lorien.lan
> Connecting to host brutus.lorien.lan, port 5201
> [  4] local 192.168.0.5 port 43654 connected to 192.168.0.36 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec   106 MBytes   885 Mbits/sec    0    220 KBytes
> [  4]   1.00-2.00   sec   108 MBytes   902 Mbits/sec    0    220 KBytes
> [  4]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0    220 KBytes
> [  4]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    220 KBytes
> [  4]   4.00-5.00   sec   112 MBytes   935 Mbits/sec    0    220 KBytes
> [  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    220 KBytes
> [  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    220 KBytes
> [  4]   7.00-8.00   sec   109 MBytes   917 Mbits/sec    0    220 KBytes
> [  4]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    220 KBytes
> [  4]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0    220 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec    0             sender
> [  4]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec                  receiver
>
> - e.g. from the start the speed is close to the max.

close, but the first interval is slower, indicating some rampup.
That's expected.

> The lack of symetry is strange - from NetBSD to W10 - full speed; from
> W10 to NetBSD - about a third... At the same time there is no
> significant difference if instead of W10 you put Linux or FreeBSD -
> both ways it is similar. And it can't be thrown at iperf3 on W10 only
> - when the server is Linux or FreeBSD, the speed is as expected.

I wouldn't expect symmetry.  A TCP connection being exercised at full
speed involves code on the sender that deals with the congestion window,
fast retransmit, etc. based on acknowledgements (and SACK).  The
receiver's code that matters is about generating acks (and there is
usually some delayed ack notion).  Similarly, the sending side of one
driver and receiving side of the other driver are being stressed.