Re: Poor TCP performance as latency increases

[Greg Troxel <gdt%lexort.com@localhost>]

Jared McNeill <jmcneill%invisible.ca@localhost> writes:

> It seems that there is a performance issue in the TCP stack, and I’m
> hoping someone else can reproduce my findings as I’m not an expert in
> this area.

The way to diagnose this is by looking at traces, and that is very hard
by hand.  graphics/xplot-devel has a script to convert tcpdump output to
xplot inptut.  It needs updates because tcpdump's output format has not
been stable, and I can send that to you privately, or you can ask me to
analyze a tcpdump.  It's best to have simultaneous dumps from both
sides.

> Simple test: Download a file from a web server. As latency between the
> client and server increases, there is a drastic reduction in
> throughput. I don’t see this with other (Linux, macOS) TCP stacks.

I wonder if there is any loss.  I would suggest "netstat -s" saved
before and after the download and diffing, to look for things you aren't
expecting.

>> 0.2ms
> berserk$ ping -c1 10.0.0.100
> PING aigis (10.0.0.100): 56 data bytes
> 64 bytes from 10.0.0.100: icmp_seq=0 ttl=64 time=0.213683 ms
> berserk$ curl -o /dev/null http://10.0.0.100/files/1Gb.dat
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 1024M  100 1024M    0     0   112M      0  0:00:09  0:00:09 --:--:—  112M
>
>> 20ms
> aigis$ ping -c1 cdn.netbsd.org
> PING cdn.netbsd.org (151.101.137.6): 56 data bytes
> 64 bytes from 151.101.137.6: seq=0 ttl=57 time=21.185 ms
> berserk$ curl -o /dev/null http://cdn.netbsd.org/pub/NetBSD/images/10.1/NetBSD-10.1-evbarm-aarch64.iso
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  261M  100  261M    0     0  21.2M      0  0:00:12  0:00:12 --:--:-- 26.7M
>
>> 30ms
> berserk$ ping -c1 bhs.proof.ovh.ca
> PING bhs.proof.ovh.ca (51.222.154.207): 56 data bytes
> 64 bytes from 51.222.154.207: icmp_seq=0 ttl=50 time=28.642797 ms
> berserk$ curl -o /dev/null https://bhs.proof.ovh.ca/files/1Gb.dat
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 1024M  100 1024M    0     0  10.5M      0  0:01:36  0:01:36 --:--:-- 9674k
>
>
> From the Linux client (on the same network, behind the same router), performance is reasonable:
>
>> 20ms
> aigis$ ping -c1 cdn.netbsd.org
> PING cdn.netbsd.org (151.101.137.6): 56 data bytes
> 64 bytes from 151.101.137.6: seq=0 ttl=57 time=21.185 ms
> aigis$ curl -o /dev/null http://cdn.netbsd.org/pub/NetBSD/images/10.1/NetBSD-10.1-evbarm-aarch64.iso
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  261M  100  261M    0     0   137M      0  0:00:01  0:00:01 --:--:--  137M
>
>
>> 30ms
> aigis$ ping -c1 bhs.proof.ovh.ca
> PING bhs.proof.ovh.ca (51.222.154.207): 56 data bytes
> 64 bytes from 51.222.154.207: seq=0 ttl=50 time=28.453 ms
> aigis$ curl -o /dev/null https://bhs.proof.ovh.ca/files/1Gb.dat
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 1024M  100 1024M    0     0  85.5M      0  0:00:11  0:00:11 --:--:— 95.7M

So not only less, but variable.

I often see most of 100 Mb/s downloads, on my theoretical 300 Mb/s pipe.

You are geting 112M which is most of 1 Gb/s locally.

I don't know what kind of access pipe you have and how the network is
between you and the download servers.   Even 10.5M is I think 84 Mb/s.