Network interface slowness

[BERTRAND Joël <joel.bertrand%systella.fr@localhost>]

	Hello,

	I have replaced my old DSL connection (2Mbps) by a professionnal
FTTH/4G one.

CISCO FTTH adapter
   |        |
   |        |
netbsd----Linux
   |
   |
  lan

NetBSD and Linux (devuan) servers are directly connected to Cisco
adapter with public IPv4 addresses. No traffic shapers, only QoS
(iptables TOS on Linux side, altqd on NetBSD side).

On Linux, I got :
- 950 Mbps downlink;
- 295 Mbps uplink.

ISP announces 1 Gbps down and 300 Mbps up.

On NetBSD (-10) or workstations on LAN:
- 400 Mbps downlink;
- 4 (!) Mbps uplink, yes, 4 Mbps.

Of course, these results were done in the same configuration (firefox on
Linux and NetBSD console and this test: https://www.nperf.com/fr/). On
NetBSD, I have tried with and without altqd.

Network configuration on NetBSD server is complex:
- wm0, wm1: bridge0 connected to two NAS (iSCSI)
- wm2: WAN access
- wm2, wm4: lagg0 connected to LAN (through a Cisco switch)
- re0: connection to Linux box (DMZ)

Kernel shows:
wm2 at pci0 dev 25 function 0, 64-bit DMA: I218 V Ethernet Connection
(rev. 0x00)
wm2: interrupting at msi4 vec 0
wm2: PCI-Express bus
wm2: 2048 words FLASH, version 0.1.4
wm2: Ethernet address 08:62:66:47:63:99
wm2: 0x6a4080<FLASH,PCIE,ASF_FIRM,AMT,WOL,EEE>
ihphy2 at wm2 phy 2: i217 10/100/1000 media interface, rev. 5
ihphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto

wm3 at pci6 dev 0 function 0, 64-bit DMA: Intel i82574L (rev. 0x00)
wm3: for TX and RX interrupting at msix6 vec 0 affinity to 1
wm3: for TX and RX interrupting at msix6 vec 1 affinity to 2
wm3: for LINK interrupting at msix6 vec 2
wm3: PCI-Express bus
wm3: 2048 words FLASH, version 1.8.0, Image Unique ID 0000ffff
wm3: ASPM L0s and L1 are disabled to workaround the errata.
wm3: Ethernet address 68:05:ca:02:b2:59
wm3: 0x224080<FLASH,PCIE,ASF_FIRM,WOL>
makphy0 at wm3 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1

wm4 at pci7 dev 0 function 0, 64-bit DMA: Intel i82574L (rev. 0x00)
wm4: for TX and RX interrupting at msix7 vec 0 affinity to 1
wm4: for TX and RX interrupting at msix7 vec 1 affinity to 2
wm4: for LINK interrupting at msix7 vec 2
wm4: PCI-Express bus
wm4: 2048 words FLASH, version 1.8.0, Image Unique ID 0000ffff
wm4: ASPM L0s and L1 are disabled to workaround the errata.
wm4: Ethernet address 68:05:ca:44:cf:30
wm4: 0x224080<FLASH,PCIE,ASF_FIRM,WOL>
makphy1 at wm4 phy 1: Marvell 88E1149 Gigabit PHY, rev. 1

I only copy wm[234] and lagg0 configuration:
wm2: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>

capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
        enabled=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
        enabled=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
        ec_capabilities=0x17<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,EEE>
        ec_enabled=0x2<VLAN_HWTAGGING>
        address: 08:62:66:47:63:99
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::a62:66ff:fe47:6399%wm2/64 flags 0 scopeid 0x3
        inet x.y.z.t/29 broadcast x.y.z.t flags 0
wm3: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>

capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0
        ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
        ec_enabled=0x2<VLAN_HWTAGGING>
        address: 68:05:ca:02:b2:59
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
wm4: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>

capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0
        ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
        ec_enabled=0x2<VLAN_HWTAGGING>
        address: 68:05:ca:02:b2:59
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        link 68:05:ca:44:cf:30
lagg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>

capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0x3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
        enabled=0x3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
        enabled=0x3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
        ec_capabilities=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
        ec_enabled=0x2<VLAN_HWTAGGING>
        laggproto loadbalance
        laggport:
                wm3 pri=32768 flags=0x1c<ACTIVE,COLLECTING,DISTRIBUTING>
                wm4 pri=32768 flags=0x1c<ACTIVE,COLLECTING,DISTRIBUTING>
        address: 68:05:ca:02:b2:59
        status: active
        inet6 fe80::6a05:caff:fe02:b259%lagg0/64 flags 0 scopeid 0x9
        inet6 2001:7a8:a8ed:10::128/64 flags 0
        inet 192.168.10.128/24 broadcast 192.168.10.255 flags 0

and I have added in /etc/sysctl.conf:

net.inet.ip.forwarding=1
kern.maxfiles=32768
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.init_win=10
net.inet.tcp.init_win_local=10
net.inet.ip.ifq.maxlen=4096
kern.ipc.semmni=1024
kern.ipc.semmns=4096
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_auto=1
net.inet6.tcp6.recvbuf_auto=1
net.inet6.tcp6.sendbuf_auto=1
kern.sbmax=8388608
net.inet.tcp.sendspace=3217968
net.inet.tcp.recvspace=3217968

	I suppose I have to tune some parameters to obtain a better throughput,
mainly on uplink.

	Help will be welcome.

	Best regards,

	JKB