Re: kern/56978: nvme hangs under very heavy loads

[Paul Goyette <paul%whooppee.com@localhost>]

The following reply was made to PR kern/56978; it has been noted by GNATS.

From: Paul Goyette <paul%whooppee.com@localhost>
To: Michael van Elst <mlelstv%serpens.de@localhost>, gnats-bugs%netbsd.org@localhost
Cc: Robert Elz <kre%munnari.OZ.AU@localhost>, mlelstv%netbsd.org@localhost
Subject: Re: kern/56978: nvme hangs under very heavy loads
Date: Wed, 7 Sep 2022 12:07:00 -0700 (PDT)

 At Michael's suggestion, I have reconfigured the problem machine
 with only two NVMe devices - one in the on-board "native" M2 slot,
 and one in a regular PCIe slot using a "simple" single-NVMe bridge
 card.
 
 The killer workload which previously was nearly certain to fail is
 now running happily!
 
 So this is likely a problem with the PCIe switch card I tried (with
 hopes of using a total of three NVMe) and/or the motherboard's
 firmware.  The failure occurred on the switch regardless of which
 NVMe slots were occupied (slot 1, 2, or both).
 
 Since this doesn't seem to be a NetBSD issue, I guess we can close
 the PR?
 
 On Fri, 2 Sep 2022, Michael van Elst wrote:
 
 > On Thu, Sep 01, 2022 at 01:09:41PM -0700, Paul Goyette wrote:
 >>> It's also unclear if the problem is in the switch or in how it is
 >>> used, so it would be helpful to see how your otherwise unchanged
 >>> system behaves.
 >>
 >> Not sure what you mean by ``otherwise unchanged''...
 >
 > Same system, same NVME M.2 cards, but without (using) the switch.
 >
 > My assumption was that you could remove one of the NVME M.2 cards (you
 > said, the issue occurs with only one slot populated) but also replace
 > the switch card with a "dumb" card with only one M.2 slot.
 >
 > If the issue still showed up, the PCIe configuration (switch/bridge/whatever)
 > would probably be unrelated.
 >
 > But if the issue were gone, then it's probably caused by some combination
 > of your CPU (with the PCIe root complex) and such a switch. I would
 > suspect some erratum in either, Linux has lots of workaounds for such
 > quirks that might help.
 >
 >
 > Greetings,
 > --
 >                                Michael van Elst
 > Internet: mlelstv%serpens.de@localhost
 >                                "A potential Snark may lurk in every tree."
 >
 > !DSPAM:631178ba170271864915606!
 >
 >
 
 +--------------------+--------------------------+----------------------+
 | Paul Goyette       | PGP Key fingerprint:     | E-mail addresses:    |
 | (Retired)          | FA29 0E3B 35AF E8AE 6651 | paul%whooppee.com@localhost    |
 | Software Developer | 0786 F758 55DE 53BA 7731 | pgoyette%netbsd.org@localhost  |
 | & Network Engineer |                          | pgoyette99%gmail.com@localhost |
 +--------------------+--------------------------+----------------------+