kern/56407: nvme(4) not working for alpha (page size 8KB) and powerpc/ibm4xx (16KB)

[rokuyama%meiji.ac.jp@localhost]

>Number:         56407
>Category:       kern
>Synopsis:       nvme(4) not working for alpha (page size 8KB) and powerpc/ibm4xx (16KB)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Sep 16 07:40:00 +0000 2021
>Originator:     Rin Okuyama
>Release:        9.99.88
>Organization:
Department of Physics, Meiji University
>Environment:
NetBSD dht 9.99.88 NetBSD 9.99.88 (DHT) #249: Mon Sep 13 21:29:49 JST 2021  rin@latipes:/sys/arch/evbppc/compile/DHT evbppc
>Description:
nvme(4) does not work for alpha (page size 8K) and powerpc/ibm4xx (16K).
dmesg for both machines are:

alpha: https://gist.github.com/rokuyama/b0482d370f9551ef2685628ad32d11c2

ibm4xx: https://gist.github.com/rokuyama/f620f0b375721e25087b574a8fa001f7

Both machines do not have native PCIe support; nvme(4) is connected via
PCI-PCIe reverse bridge. This may be a cause of the problem, but I suspect
another scenario.

As indicated dmesg for ibm4xx above, KASSERT (added to my local tree):

----
#define NVME_DONE_CHECK(done)   KASSERTMSG(	\
    (done) == nvme_ns_io_done ||		\
    (done) == nvme_ns_sync_done ||		\
    (done) == nvme_getcache_done ||		\
    (done) == nvme_setcache_done ||		\
    (done) == nvme_pt_done ||			\
    (done) == nvme_poll_done ||			\
    (done) == nvme_empty_done,			\
    "unknown done callback %p", (done));

	NVME_DONE_CHECK(ccb->ccb_done);
----

is triggered. This suggests that DMA does not work correctly for CCB.

As far as I know, all platforms with working nvme(4) have 4K pages: x86,
aarch64, arm, and powerpc/oea and booke. Therefore, I guess that something is
wrong in current codes for page size > 4K.

I've examined MPSMAX and MPSMIN for some products. For all the devices I
have, MPSMIN is 4K. For some products, MPSMAX is also 4K. But, MPSMAX is
8K for Intel 760p, HP EX920, and Kingston KC2500. However, these devices
do not work for alpha, unfortunately.
>How-To-Repeat:
Try to use nvme(4) for alpha or powerpc/ibm4xx.
>Fix:
N/A