Network changes between 4.x and 5.1RC3 ?

[Jorgen Lundman <lundman%lundman.net@localhost>]

We have run NetBSD 4 on a server for over a year and recently changed to NetBSD 
5.0RC3. But we have been experiencing some network issues. Apart from the kernel 
and userland upgrade, nothing has chanced with hardware, setup, routing etc.

We appear to experiencing network pauses, general sluggishness and sometimes 
transfers just appear to stall. Since its main objective is to be an anonymous 
FTP mirror, this is peculiar.

I know it is fairly vague, but perhaps it is well known that some part of the IP 
stack changed, that I could look into.


Many large transfers look like this, in the last few packets:

sndr: NetBSD 5.0.2
rcvr: NetBSD 5.1RC3

04:31:28.211334 IP rcvr.443 > sndr.52559: S 1684671976:1684671976(0) win 65535 
<mss 1460,nop,wscale 8,sackOK,nop,nop,nop,nop,timestamp 1 0>

04:31:28.518422 IP sndr.52559 > rcvr.443: S 3182409332:3182409332(0) ack 
1684671977 win 65535 <mss 1460,nop,wscale 8,nop,nop,timestamp 1 1,sackOK,nop,nop>

04:31:28.518447 IP rcvr.443 > sndr.52559: . ack 1 win 1260 <nop,nop,timestamp 2 
1>

04:31:28.804494 IP sndr.52559 > rcvr.443: . ack 1 win 1785 <nop,nop,timestamp 2 
2>

lots, and lots of data, until the end:

05:11:48.972870 IP rcvr.443 > sndr.52559: . ack 522788159 win 3041 
<nop,nop,timestamp 4841 4815>

05:11:48.973013 IP sndr.52559 > rcvr.443: . 522788159:522789607(1448) ack 1115 
win 1785 <nop,nop,timestamp 4815 4840>

05:11:48.973156 IP sndr.52559 > rcvr.443: FP 522789607:522790463(856) ack 1115 
win 1785 <nop,nop,timestamp 4815 4840>

05:11:48.973169 IP rcvr.443 > sndr.52559: . ack 522790464 win 3041 
<nop,nop,timestamp 4841 4815>

05:11:48.974764 IP rcvr.443 > sndr.52559: P 1115:1152(37) ack 522790464 win 3052 
<nop,nop,timestamp 4841 4815>

05:11:48.974802 IP rcvr.443 > sndr.52559: F 1152:1152(0) ack 522790464 win 3052 
<nop,nop,timestamp 4841 4815>

05:11:49.252555 IP sndr.52559 > rcvr.443: R 3705199796:3705199796(0) win 0

05:11:49.252558 IP sndr.52559 > rcvr.443: R 3705199796:3705199796(0) win 0


and it pretty much sits there forever.



Both old 4.x and new 5.1RC3 server settings are default, with only these values:

options         NMBCLUSTERS=32768

kern.sbmax=8388608
net.inet.tcp.sendspace=321796
net.inet.tcp.recvspace=321796
net.inet.tcp.init_win=8


I have since also tried a few more settings on NetBSD5.1RC3 to see if they would 
help, but it would appear to have made little difference. Current sysctl output is:

net.inet.tcp.rfc1323 = 1
net.inet.tcp.sendspace = 321796
net.inet.tcp.recvspace = 321796
net.inet.tcp.mssdflt = 536
net.inet.tcp.syn_cache_limit = 10255
net.inet.tcp.syn_bucket_limit = 105
net.inet.tcp.init_win = 8
net.inet.tcp.mss_ifmtu = 0
net.inet.tcp.sack.enable = 1
net.inet.tcp.sack.maxholes = 32
net.inet.tcp.sack.globalmaxholes = 1024
net.inet.tcp.sack.globalholes = 1
net.inet.tcp.win_scale = 1
net.inet.tcp.timestamps = 1
net.inet.tcp.compat_42 = 0
net.inet.tcp.cwm = 0
net.inet.tcp.cwm_burstsize = 4
net.inet.tcp.ack_on_push = 1
net.inet.tcp.keepidle = 14400
net.inet.tcp.keepintvl = 150
net.inet.tcp.keepcnt = 8
net.inet.tcp.slowhz = 2
net.inet.tcp.log_refused = 0
net.inet.tcp.rstppslimit = 100
net.inet.tcp.delack_ticks = 20
net.inet.tcp.init_win_local = 4
net.inet.tcp.do_loopback_cksum = 0
net.inet.tcp.msl = 30
net.inet.tcp.minmss = 216
net.inet.tcp.congctl.available = reno newreno
net.inet.tcp.congctl.selected = newreno
net.inet.tcp.keepinit = 150
net.inet.tcp.recvbuf_auto = 1
net.inet.tcp.recvbuf_inc = 65536
net.inet.tcp.recvbuf_max = 16777216
net.inet.tcp.sendbuf_auto = 1
net.inet.tcp.sendbuf_inc = 65536
net.inet.tcp.sendbuf_max = 16777216
net.inet.tcp.ecn.enable = 0
net.inet.tcp.ecn.maxretries = 1
net.inet.tcp.iss_hash = 0
net.inet.tcp.abc.enable = 1
net.inet.tcp.abc.aggressive = 1



--
Jorgen Lundman       | <lundman%lundman.net@localhost>
Unix Administrator   | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo    | +81 (0)90-5578-8500          (cell)
Japan                | +81 (0)3 -3375-1767          (home)