Re: Weird network performance problem

[Jason Mitchell <jmitchel%bigjar.com@localhost>]

> On Jan 19, 2020, at 12:01 PM, Greg Troxel <gdt%lexort.com@localhost> wrote:
> 
> Chavdar Ivanov <ci4ic4%gmail.com@localhost> writes:
> 
>>>  It looks like you are using vlan support on Y.  Try without also.
>> 
>> That may be something to look at. This is my NVMM host as well, every
>> boot I recreate tap[0..5] for use by the NVMM guests (but the tests
>> were done without any of them running).
>> 
>> I am not using vlans deliberately - the switch upstairs is a dumb one,
>> although the one downstaris is managed and has (unusued at the moment)
>> vlan support. The interfaces are created simply with /etc/ifconfig.wm0
>> - just 'inet 192.168.0.29 netmask 255.255.255.0 up description "My
>> LAN"' and /etc/ifconfig.bridge0 -
> 
> I meant that hardware vlan support was enabled.  I really doubt this is
> the issue, but it seems easy to try the easy things.
> 
> Also, I forgot my other usual advice.
> 
>  $ netstat -s > BEFORE
>  $ # do iperf
>  $ netstat -s > AFTER
>  $ diff -u BEFORE AFTER
> 
> to find out which counters that you didn't even know existed are
> changing  in perhaps interesting ways.
> 
>> From the XCP-NG host to the NetBSD laptop:
>> 
>> $ iperf3 -c ymir.lorien.lan
>> Connecting to host ymir.lorien.lan, port 5201
>> [  4] local 192.168.0.5 port 36036 connected to 192.168.0.29 port 5201
>> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
>> [  4]   0.00-1.00   sec  45.9 MBytes   385 Mbits/sec    0   66.5 KBytes
>> [  4]   1.00-2.00   sec  64.2 MBytes   539 Mbits/sec    0    100 KBytes
>> [  4]   2.00-3.00   sec  81.3 MBytes   682 Mbits/sec    0    132 KBytes
>> [  4]   3.00-4.00   sec  99.4 MBytes   834 Mbits/sec    0    163 KBytes
>> [  4]   4.00-5.00   sec   109 MBytes   911 Mbits/sec    0    205 KBytes
>> [  4]   5.00-6.00   sec   111 MBytes   928 Mbits/sec    0    205 KBytes
>> [  4]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    0    205 KBytes
>> [  4]   7.00-8.00   sec   111 MBytes   932 Mbits/sec    0    205 KBytes
>> [  4]   8.00-9.00   sec   111 MBytes   930 Mbits/sec    0    205 KBytes
>> [  4]   9.00-10.00  sec   111 MBytes   932 Mbits/sec    0    205 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval           Transfer     Bandwidth       Retr
>> [  4]   0.00-10.00  sec   954 MBytes   800 Mbits/sec    0             sender
>> [  4]   0.00-10.00  sec   953 MBytes   800 Mbits/sec                  receiver
>> 
>> Starts a bit slower, but after the fourth interval reaches along the maximum.
> 
> That's just 1s periods within the same TCP connection.  That is more
> expected than subsequent TCP connections.  But, this is a clue of
> something to perhaps change.  NetBSD has defaults for send and receive
> buffer sizes, and some notion of auto tuning.  I am not really clear
> where iperf3 is getting those "Cwnd" values, but perhaps it is able to
> obtain them from the Linux kernel?
> 
> On some machines I have
> 
> net.inet.tcp.sendspace=131072
> net.inet.tcp.recvspace=131072
> net.inet6.tcp6.sendspace=131072
> net.inet6.tcp6.recvspace=131072
> 
> net.inet.tcp.recvbuf_auto=0
> net.inet.tcp.sendbuf_auto=0
> net.inet6.tcp6.recvbuf_auto=0
> net.inet6.tcp6.sendbuf_auto=0
> 
> which may not be the right thing, but at one point I had trouble with
> the auto stuff not rampning up fast enough.
> 
>> When the server is the B laptop running W10, I get:
>> 
>> $ iperf3 -c brutus.lorien.lan
>> Connecting to host brutus.lorien.lan, port 5201
>> [  4] local 192.168.0.5 port 43654 connected to 192.168.0.36 port 5201
>> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
>> [  4]   0.00-1.00   sec   106 MBytes   885 Mbits/sec    0    220 KBytes
>> [  4]   1.00-2.00   sec   108 MBytes   902 Mbits/sec    0    220 KBytes
>> [  4]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0    220 KBytes
>> [  4]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    220 KBytes
>> [  4]   4.00-5.00   sec   112 MBytes   935 Mbits/sec    0    220 KBytes
>> [  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    220 KBytes
>> [  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    220 KBytes
>> [  4]   7.00-8.00   sec   109 MBytes   917 Mbits/sec    0    220 KBytes
>> [  4]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    220 KBytes
>> [  4]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0    220 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval           Transfer     Bandwidth       Retr
>> [  4]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec    0             sender
>> [  4]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec                  receiver
>> 
>> - e.g. from the start the speed is close to the max.
> 
> close, but the first interval is slower, indicating some rampup.
> That's expected.
> 
>> The lack of symetry is strange - from NetBSD to W10 - full speed; from
>> W10 to NetBSD - about a third... At the same time there is no
>> significant difference if instead of W10 you put Linux or FreeBSD -
>> both ways it is similar. And it can't be thrown at iperf3 on W10 only
>> - when the server is Linux or FreeBSD, the speed is as expected.
> 
> I wouldn't expect symmetry.  A TCP connection being exercised at full
> speed involves code on the sender that deals with the congestion window,
> fast retransmit, etc. based on acknowledgements (and SACK).  The
> receiver's code that matters is about generating acks (and there is
> usually some delayed ack notion).  Similarly, the sending side of one
> driver and receiving side of the other driver are being stressed.

Two things:

First, Powerline adapters only get 1/7th the advertised speed. They send “1Gigabit” of data, but they’re really sending ~130Mbit of the same data 7 times because of how noisy power cabling is.

Second, try adding -l 9000 to your iperf3 tests. The Linux distributions I’ve used have this as the default and I’ve noticed big speed improvements doing this, but I have no clue why it works.

Thanks,

Jason M.

Sent from my iPhone