Re: Dom0 detects no carrier on ixg

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

Hi.

On 2022/12/15 9:44, Brook Milligan wrote:
> 
>> On Dec 14, 2022, at 7:54 AM, Manuel Bouyer <bouyer%antioche.eu.org@localhost> wrote:
>>
>> On Wed, Dec 14, 2022 at 07:40:53AM -0700, Brook Milligan wrote:
>>>
>>>> On Dec 14, 2022, at 1:39 AM, Manuel Bouyer <bouyer%antioche.eu.org@localhost> wrote:
>>>>
>>>> 55 is ENOBUFS; you don't have enough memory, or more probably enough
>>>> mbuf clusters
>>>>
>>>> NMBCLUSTERS is computed automatically based on RAM amount, but for a
>>>> dom0 (which typically has a small portion of the physical RAM allocated),
>>>> it ends up too low for modern NIC.
>>>>
>>>> I use
>>>> options NMBCLUSTERS=262144
> 
> OK, I’ve increased this to 4M (4,166,656) and RAM to 8GB; no more ENOBUFS errors.
> 
> These seem like quite high values.  Do I just fiddle to see what is required?  Is there any rule of thumb for figuring this out?

Some info:

(For simplicity of explanation, the network interface here is assumed to be
that of a PCIe-based Ethernet.)

 Almost all Ethernet drivers setup at least one RX queue. Number of entries
in an RX queue is driver dependent. 1Gbps Ethernet drivers usually uses
256 or 512 entries per queue. 10Gbps or more high speed Ethernet drivers
uses 512-2048 entries. Usually, when initializing RX ring, one mbuf is assigned
to each entry.

 Modern network drivers use multiple RX queues to process packets in parallel.
The number of queues are (basically) calculated by
	MIN(ncpu, (AVAILABLE_NUMBER_OF_MSIX_VECTOR_OF_THE_CHIP - 1)

So,

 - High speed Ethernet drivers use more mbufs.
 - Ethernet drivers using with multiqueue use more mbufs.
 - If a system has many CPUs, Ethernet drivers using with multiqueue use more mbufs.
 - If a system has multiple Ethernet interfaces using with multiqueue use more mbufs.
	



> Thanks for your help.
> 
> Cheers,
> Brook
> 

-- 
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)