Re: Gateway server experiencing degraded performance

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

On Mon, 16 May 2022, Andrew K Adams wrote:
> I've looked at traffic with tcpdump, but there is so much in both
> scenarios that I can't really say that one service, or a
> packet-of-death is triggering the degraded state.  But thinking in
> aggregate, if there was a way to monitor the average packets/sec the
> gateway is forwarding, in both scenarios, I'd at least know for
> certain that it is or is not my gateway (I guess it's conceivable that
> by rebooting I'm tweaking state in Comcast's next-hop during DHCP
> negotiation or something ...)

net/vnstat in pkgsrc might be useful for this.

> > > Hi, I’ve noticed a problem with my NetBSD server that I really could
> > > use some help with.  The server is acting as my home router (gateway)
> > > in ‘pass-through’ mode with Comcast Xfinity as the service provider.
> > > The only functions/services enabled on the server are: ip-forwarding,
> > > dhcp-client, and NTP.   And on the internal-facing lan: NAT, DHCP, and
> > > SSH.  The internal network supports 5+ wired hosts & 10+ wireless
> > > devices.  I currently have the 1Gig option with Comcast and when
> > > things are operating as expected, I see from an internal (wired) host
> > > 750mbs of throughput according to Comcast’s Xfinity speed test
> > > website.  The problem is that performance doesn’t last.  In anywhere
> > > from 3 weeks to 24 hours, my network performance (again, using the
> > > same wired host measured by Xfinity’s speed test) will drop to ~20mbs.
> > > And the change is clearly observable in all networking applications,
> > > e.g., Netflix.  I’m querying here, though, because the fix I’ve found
> > > is to reboot the server.  Upon reboot, I immediately go back to
> > > 750mbs.  Hence, I’d like to figure out how to troubleshoot what on my
> > > server is (eventually) degrading my network performance.
> > >
> > > The server is relatively new; I built it using an ASUS Mini ITX with
> > > two onboard GigE ethernet ports.  Here’s some relevant dmesg output:
> > >
> > > [     1.000000] NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020
> > > [     1.000000]
> > > mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/i386/compile/GENERIC
> > > [     1.000000] total memory = 3457 MB
> > > [     1.000000] avail memory = 3380 MB
> > >
> > > [     1.025286] wm0 at pci0 dev 25 function 0: I217 LM Ethernet
> > > Connection (rev. 0x05)
> > > [     1.025286] wm0: interrupting at msi2 vec 0
> > > [     1.025286] wm0: PCI-Express bus
> > > [     1.025286] wm0: 2048 words FLASH, version 0.13.4
> > > [     1.025286] wm0: 0x6a4480<FLASH,IOH_VALID,PCIE,ASF_FIRM,AMT,WOL,EEE>
> > >
> > > [     1.025286] re0 at pci3 dev 0 function 0: RealTek 8168/8111 PCIe
> > > Gigabit Ethernet (rev. 0x0c)
> > > [     1.025286] re0: interrupting at msix4 vec 0
> > > [     1.025286] re0: using 256 tx descriptors
> > >
> > >
> > > And from ifconfig (note, I’ve removed my IP & MAC address from the output):
> > >
> > > iquitos# ifconfig -a
> > > wm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > >         capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
> > >         capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
> > >         capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
> > >         enabled=0
> > >         ec_capabilities=17<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,EEE>
> > >         ec_enabled=2<VLAN_HWTAGGING>
> > >         media: Ethernet autoselect (1000baseT
> > > full-duplex,flowcontrol,rxpause,txpause)
> > >         status: active
> > >         inet X.X.X.X/20 broadcast 255.255.255.255 flags 0x0
> > >         Inet6 X%wm0/64 flags 0x0 scopeid 0x1
> > > re0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > >         capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
> > >         capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
> > >         enabled=0
> > >         ec_capabilities=3<VLAN_MTU,VLAN_HWTAGGING>
> > >         ec_enabled=0
> > >         media: Ethernet autoselect (1000baseT full-duplex)
> > >         status: active
> > >         inet 10.19.34.1/24 broadcast 10.19.34.255 flags 0x0
> > >         inet6 fe80::728b:cdff:febc:831b%re0/64 flags 0x0 scopeid 0x2
> > > lo0: flags=0x8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33176
> > >         inet 127.0.0.1/8 flags 0x0
> > >         inet6 ::1/128 flags 0x20<NODAD>
> > >         inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x3
> > >
> > >
> > > I’m not sure what other information would be useful, just let me know,
> > > and thank you in advance for help you can give me!
> >
> > Just a question, not a requirement, but have you looked at Wireshark
> > traces comparing good performance vs. bad performance?
> >
> > It's just another way to find possibly find a problem. I'm sure
> > someone else will have better ideas.
> >
> > Andy