tech-net archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

NetBSD 5.1 TCP performance issue (lots of ACK)



Hello,
I've been playing with glusterfs a bit today, and found some performance
differences between NetBSD and linux, which I tracked down to our TCP
stack. Basically, between a NetBSD/linux pair, performances are much
better than between 2 NetBSD hosts. It doesn't matter if linux is client
or server, so this point some issue outside of glusterfs.

So I have done some packet capture and found a strange TCP behavior between
2 NetBSD hosts.

The setup: 192.168.1.2 is a NetBSD 5.1 glusterfs server, 192.168.1.1 is a
NetBSD 5.1 glusterfs client and 192.168.1.3 is a linux (RHEL6) glusterfs
client. The clients read a 640Mb file from the server (with
dd if=file of=/dev/null bs=64k). All 3 hosts are strictly identical
hardware (same CPUs, ram, motherboard, gigabit network adapter and hard disk).
The linux client can read at 95MB/s from the NetBSD server, the NetBSD
client only 50MB/s. (But in other tests, a NetBSD client can read at 90MB/s
out of a linux server, no neither the NetBSD server nor the NetBSD client
is the bottleneck in a NetBSD/NetBSD setup).

Attached are the tcptrace outputs for both client. The problem is, the
NetBSD client is sending 242873 packets to the server for the file read,
while the linux client sends only 34581 packets), so I suspect there's
something wrong with our TCP ack code (it looks like we ack way too often).

The relevant part of sysctl.conf of the client is:
net.inet.tcp.sendbuf_auto=0
net.inet.tcp.recvbuf_auto=0
kern.sbmax=4194304
net.inet.tcp.sendbuf_max=1048576
net.inet.tcp.recvbuf_max=1048576
net.inet.tcp.sendspace=524288
net.inet.tcp.recvspace=524288
net.inet.tcp.abc.enable=0
net.inet.ip.ifq.maxlen=512

Any idea of a sysctl setup (or something else) that could help ?

I actually suspect it's also the reason why I see a lower data rate
over my ADSL connection when downloading from ftp.fr.netbsd.org than
when talking do some linux server ...

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--
================================
TCP connection 3:
        host e:        192.168.1.3:1020
        host f:        192.168.1.2:24009
        complete conn: yes
        first packet:  Mon Oct 17 16:53:29.631487 2011
        last packet:   Mon Oct 17 16:53:45.620919 2011
        elapsed time:  0:00:15.989432
        total packets: 121632
        filename:      re.pcap
   e->f:                              f->e:
     total packets:         34581           total packets:         87051      
     ack pkts sent:         34580           ack pkts sent:         87051      
     pure acks sent:        29555           pure acks sent:         1835      
     sack pkts sent:            0           sack pkts sent:            0      
     dsack pkts sent:           0           dsack pkts sent:           0      
     max sack blks/ack:         0           max sack blks/ack:         0      
     unique bytes sent:    845208           unique bytes sent: 121406760      
     actual data pkts:       5024           actual data pkts:      85214      
     actual data bytes:    845208           actual data bytes: 121406760      
     rexmt data pkts:           0           rexmt data pkts:           0      
     rexmt data bytes:          0           rexmt data bytes:          0      
     zwnd probe pkts:           0           zwnd probe pkts:           0      
     zwnd probe bytes:          0           zwnd probe bytes:          0      
     outoforder pkts:           0           outoforder pkts:           0      
     pushed data pkts:       5024           pushed data pkts:       3411      
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1      
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y      
     adv wind scale:            7           adv wind scale:            7      
     req sack:                  Y           req sack:                  Y      
     sacks sent:                0           sacks sent:                0      
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts 
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:           424 bytes     max segm size:          1448 bytes
     min segm size:           132 bytes     min segm size:             4 bytes
     avg segm size:           168 bytes     avg segm size:          1424 bytes
     max win adv:          195200 bytes     max win adv:          525568 bytes
     min win adv:            5888 bytes     min win adv:          524928 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:          188284 bytes     avg win adv:          525482 bytes
     max owin:                841 bytes     max owin:             195197 bytes
     min non-zero owin:         1 bytes     min non-zero owin:         1 bytes
     avg owin:                 53 bytes     avg owin:             101337 bytes
     wavg owin:                37 bytes     wavg owin:             54793 bytes
     initial window:          140 bytes     initial window:          164 bytes
     initial window:            1 pkts      initial window:            1 pkts 
     ttl stream length:    845208 bytes     ttl stream length: 656063384 bytes
     missed data:               0 bytes     missed data:       534656624 bytes
     truncated data:       694488 bytes     truncated data:    118850660 bytes
     truncated packets:      5024 pkts      truncated packets:     85180 pkts 
     data xmit time:       12.909 secs      data xmit time:       12.909 secs 
     idletime max:         5634.8 ms        idletime max:         5675.2 ms   
     throughput:            52860 Bps       throughput:          7592938 Bps  

     RTT samples:            2952           RTT samples:            6551      
     RTT min:                 0.0 ms        RTT min:                 0.1 ms   
     RTT max:                 0.9 ms        RTT max:                40.1 ms   
     RTT avg:                 0.1 ms        RTT avg:                 1.1 ms   
     RTT stdev:               0.2 ms        RTT stdev:               0.8 ms   

     RTT from 3WHS:           0.0 ms        RTT from 3WHS:           0.8 ms   

     RTT full_sz smpls:         1           RTT full_sz smpls:      5239      
     RTT full_sz min:         0.2 ms        RTT full_sz min:         0.1 ms   
     RTT full_sz max:         0.2 ms        RTT full_sz max:         5.9 ms   
     RTT full_sz avg:         0.2 ms        RTT full_sz avg:         1.0 ms   
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.4 ms   

     post-loss acks:            0           post-loss acks:            0      
     segs cum acked:         2074           segs cum acked:        78665      
     duplicate acks:            1           duplicate acks:            1      
     triple dupacks:            0           triple dupacks:            0      
     max # retrans:             0           max # retrans:             0      
     min retr time:           0.0 ms        min retr time:           0.0 ms   
     max retr time:           0.0 ms        max retr time:           0.0 ms   
     avg retr time:           0.0 ms        avg retr time:           0.0 ms   
     sdv retr time:           0.0 ms        sdv retr time:           0.0 ms   
TCP connection 5:
        host i:        192.168.1.1:1021
        host j:        192.168.1.2:24009
        complete conn: yes
        first packet:  Mon Oct 17 17:42:46.460094 2011
        last packet:   Mon Oct 17 17:42:59.649114 2011
        elapsed time:  0:00:13.189019
        total packets: 342816
        filename:      xen1.3.pcap
   i->j:                              j->i:
     total packets:        242873           total packets:         99943      
     ack pkts sent:        242872           ack pkts sent:         99943      
     pure acks sent:       237850           pure acks sent:         2398      
     sack pkts sent:         1316           sack pkts sent:            0      
     dsack pkts sent:           0           dsack pkts sent:           0      
     max sack blks/ack:         1           max sack blks/ack:         0      
     unique bytes sent:    844892           unique bytes sent: 138607868      
     actual data pkts:       5021           actual data pkts:      97543      
     actual data bytes:    844892           actual data bytes: 138683996      
     rexmt data pkts:           0           rexmt data pkts:          54      
     rexmt data bytes:          0           rexmt data bytes:      76128      
     zwnd probe pkts:           0           zwnd probe pkts:           0      
     zwnd probe bytes:          0           zwnd probe bytes:          0      
     outoforder pkts:           0           outoforder pkts:          96      
     pushed data pkts:       5021           pushed data pkts:       3683      
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1      
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y      
     adv wind scale:            7           adv wind scale:            7      
     req sack:                  Y           req sack:                  Y      
     sacks sent:             1316           sacks sent:                0      
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts 
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:           408 bytes     max segm size:          1448 bytes
     min segm size:           132 bytes     min segm size:            20 bytes
     avg segm size:           168 bytes     avg segm size:          1421 bytes
     max win adv:          525568 bytes     max win adv:          525568 bytes
     min win adv:           28160 bytes     min win adv:          524160 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:          466678 bytes     avg win adv:          525530 bytes
     max owin:               1345 bytes     max owin:             521953 bytes
     min non-zero owin:         1 bytes     min non-zero owin:         1 bytes
     avg owin:                 15 bytes     avg owin:             128037 bytes
     wavg owin:               117 bytes     wavg owin:            196253 bytes
     initial window:          140 bytes     initial window:          164 bytes
     initial window:            1 pkts      initial window:            1 pkts 
     ttl stream length:    844892 bytes     ttl stream length: 656063700 bytes
     missed data:               0 bytes     missed data:       517455832 bytes
     truncated data:       694262 bytes     truncated data:    135757788 bytes
     truncated packets:      5021 pkts      truncated packets:     97532 pkts 
     data xmit time:       13.060 secs      data xmit time:       13.060 secs 
     idletime max:          253.4 ms        idletime max:          256.1 ms   
     throughput:            64060 Bps       throughput:         10509338 Bps  

     RTT samples:            2572           RTT samples:           49184      
     RTT min:                 0.0 ms        RTT min:                 0.5 ms   
     RTT max:                 0.8 ms        RTT max:               116.6 ms   
     RTT avg:                 0.1 ms        RTT avg:                 1.8 ms   
     RTT stdev:               0.1 ms        RTT stdev:               0.9 ms   

     RTT from 3WHS:           0.0 ms        RTT from 3WHS:           1.5 ms   

     RTT full_sz smpls:         1           RTT full_sz smpls:     46752      
     RTT full_sz min:         0.2 ms        RTT full_sz min:         0.5 ms   
     RTT full_sz max:         0.2 ms        RTT full_sz max:         7.0 ms   
     RTT full_sz avg:         0.2 ms        RTT full_sz avg:         1.8 ms   
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.7 ms   

     post-loss acks:            0           post-loss acks:           31      
          For the following 5 RTT statistics, only ACKs for
          multiply-transmitted segments (ambiguous ACKs) were
          considered.  Times are taken from the last instance
          of a segment.
     ambiguous acks:            0           ambiguous acks:           44      
     RTT min (last):          0.0 ms        RTT min (last):          0.7 ms   
     RTT max (last):          0.0 ms        RTT max (last):          2.3 ms   
     RTT avg (last):          0.0 ms        RTT avg (last):          1.3 ms   
     RTT sdv (last):          0.0 ms        RTT sdv (last):          0.4 ms   
     segs cum acked:         2451           segs cum acked:        48234      
     duplicate acks:            1           duplicate acks:          418      
     triple dupacks:            0           triple dupacks:            8      
     max # retrans:             0           max # retrans:             1      
     min retr time:           0.0 ms        min retr time:           4.2 ms   
     max retr time:           0.0 ms        max retr time:          43.4 ms   
     avg retr time:           0.0 ms        avg retr time:          17.7 ms   
     sdv retr time:           0.0 ms        sdv retr time:          10.0 ms   


Home | Main Index | Thread Index | Old Index