CVS commit: src

["SAITOH Masanobu" <msaitoh%netbsd.org@localhost>]

Module Name:    src
Committed By:   msaitoh
Date:           Tue Mar  9 10:03:18 UTC 2021

Modified Files:
        src/share/man/man4: ixg.4 ixv.4
        src/sys/dev/pci: files.pci
        src/sys/dev/pci/ixgbe: ix_txrx.c ixgbe.c ixgbe.h ixgbe_netbsd.h ixv.c

Log Message:
Modify some parameters to reduce packet dropping.

 - Background: ixgbe doesn't use common MCLGET() interface and use the
   driver specific cluster allocation mechanism (jcl). The cluster is
   pre-allocated with a fixed number and the current number per queue
   is num_rx_desc * 2 (2048*2=4096). It's too small. It also has a problem
   that the max length of the pcq which is used in the TX path is big
   (4096). Example:

    100M <----- [ixg0                                      ixg1] <----- 1G
                2048 TX descs <--- 4096 pcqs <---- 2048 RX descs

   If a machine forwards a traffic from 1G interface to 100M interface,
   It would require 2048+4096+2048=8192 descriptors, but the current number
   is 2048*2=4096. It's too small. Even if the both interface's link speed
   is the same and only small number of packet is queued in the pcq, 4096
   jcl is small because 2048(RX)+TX(2048)=4096. If jcl is exhausted, not only
   forwarding from ixg1 to ixg0 is dropped, but also another forwarding path
   from ixg1 to another interface(e.g. wm0) is also dropped. Sockets also
   queue packets, so if a lot of sockets are used and/or a socket buffer
   size is changed to bigger one, it'll also become a problem. If the jcl
   is exhausted, evcnt(9) counter "ixgX qY Rx no jumbo mbuf" is incremented.
   Example:
     vmstat -ev | grep ixg1 | grep "no jumbo"
     ixg1 q0 Rx no jumbo mbuf                           0     0 misc
     ixg1 q1 Rx no jumbo mbuf                           0     0 misc
     ixg1 q2 Rx no jumbo mbuf                      141326     0 misc
     ixg1 q3 Rx no jumbo mbuf                           0     0 misc

 - To solve this problem:
   - Add new config parameter IXGBE_JCLNUM_MULTI and set the default to 3
     (2048 * 3). The minimum number is 2. The total number of jcl per queue
     is available with hw.ixgN.num_jcl_per_queue sysctl.
   - Reduce the max length of the pcq() which is used in the TX path from
     4096 to 2048.

 - Reviewed by knakahara@ and ozaki-r@.

 - TODO: Use MCLGET().


To generate a diff of this commit:
cvs rdiff -u -r1.12 -r1.13 src/share/man/man4/ixg.4
cvs rdiff -u -r1.5 -r1.6 src/share/man/man4/ixv.4
cvs rdiff -u -r1.435 -r1.436 src/sys/dev/pci/files.pci
cvs rdiff -u -r1.66 -r1.67 src/sys/dev/pci/ixgbe/ix_txrx.c
cvs rdiff -u -r1.278 -r1.279 src/sys/dev/pci/ixgbe/ixgbe.c
cvs rdiff -u -r1.74 -r1.75 src/sys/dev/pci/ixgbe/ixgbe.h
cvs rdiff -u -r1.11 -r1.12 src/sys/dev/pci/ixgbe/ixgbe_netbsd.h
cvs rdiff -u -r1.154 -r1.155 src/sys/dev/pci/ixgbe/ixv.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.