re: port-evbarm/57614: Kernel panic rebooting a Pine64 RockPro64 with netbsd10.0_beta; successful after panic reboot.

[joelp%sdf.org@localhost]

I have an APC UPS attached to a USB2 port (and running apcupsd) when I see
the panic on boot.

I do not see a panic when I attach the UPS to the TypeA USB3 port. I
rebooted several times to be sure -- no panic.

I've been unable to get any more information from the debugger. I first
tried without DDB and DDB_ONPANIC=1, and about every 2nd or 3rd time, the
debugger would break, but 'show kernhist usbhist' did not give any more
details.

Looking at the docs, I tried increasing the value of hw.usb.debug, etc,
sysctl settings but that didn't help either.

I then added DDB and DDB_ONPANIC=1 options and that kernel never landed in
the debugger after several manual reboot attempts. It does still panic and
reboots (see below).

Also, per the docs, it mentions I can use boot netbsd -d to start in ddb;
however I'm never able to continue running under ddb:

[   1.0000000] total memory = 3909 MB
[   1.0000000] avail memory = 3768 MB
[   1.0000000] entropy: ready
[   1.0000000] Entering DDB...
Stopped in pid 0.0 (system) at  netbsd:cpu_Debugger+0xc:        ldp    
x29, x30
, [sp],#16
db{0}> continue

After that continue, there are several more panics, then it freezes. I'm
probably missing something here, sorry, my kernel debugging background is
Windows unfortunately :)

Is there any more config or kernel options I can include? Is there
something more I need to do after boot netbsd -d?

Here's the panic that won't break into debugger with DDB set:

Waiting for duplicate address detection to finish...
Starting dhcpcd.
[   5.0229996] panic: kernel debugging assertion
"usb_valid_block_p(f->ufd_block, &usb_blk_fraglist)" failed: file
"/lab/src-netbsd10/src/sys/dev/usb/usb_mem.c", line 303 usb_allocmem: usb
frag 0xffffc001553cf080: unknown block pointer 0x0
[   5.0229996] cpu2: Begin traceback...
[   5.0229996] trace fp ffffc001552cca60
[   5.0229996] fp ffffc001552cca90 vpanic() at ffffc000005b8768
netbsd:vpanic+0x178
[   5.0229996] fp ffffc001552ccaf0 kern_assert() at ffffc00000851588
netbsd:kern_assert+0x58
[   5.0330044] fp ffffc001552ccb80 usb_allocmem() at ffffc00000197a4c
netbsd:usb_allocmem+0x5ac
[   5.0330044] fp ffffc001552ccc00 ohci_open() at ffffc0000023f668
netbsd:ohci_open+0x198
[   5.0330044] fp ffffc001552ccc50 usbd_setup_pipe_flags() at
ffffc00000193670 netbsd:usbd_setup_pipe_flags+0x1a0
[   5.0330044] fp ffffc001552cccc0 usbd_new_device() at ffffc000001956a4
netbsd:usbd_new_device+0x744
[   5.0430010] fp ffffc001552ccd60 uhub_explore() at ffffc0000019b200
netbsd:uhub_explore+0x2d0
[   5.0430010] fp ffffc001552cce20 usb_discover() at ffffc000001846a0
netbsd:usb_discover+0x70
[   5.0430010] fp ffffc001552cce70 usb_event_thread() at ffffc00000184988
netbsd:usb_event_thread+0xc8
[   5.0430010] tf ffffc001552cced0 el0_trap() at ffffc000000b6ff0
netbsd:el1_trap_exit+0x68
[   5.0530002] cpu2: End traceback...
[   5.0530002] rebooting...

Here are my changes to the kernel config (copied from GENERIC64):

#options    DIAGNOSTIC  # internal consistency checks
options     DDB
options     DDB_ONPANIC=1
options     DEBUG
options     USB_DEBUG
options     UHUB_DEBUG
options     XHCI_DEBUG
options     EHCI_DEBUG
#options    LOCKDEBUG

Thanks - Joel

> do you have devices plugged in to the usb2 (ehci) port?  if possible,
> can you use the usb3 (xhci) port only and see if the problem occurs?
>
> i've seen this problem (at least the syncmem version of it) irregularly
> for a while on rk3399 (pinebookpro mostly, not very often on rockpro64),
> but i haven't yet figured out what causes it.
>
> i don't think there's a race condition related fsck, while it normally
> happens at reboot, i've seen it after being up, and then later inserting
> a usb device and it crashed.
>
> (i thought i'd logged a PR about it, but i can't find it.)
>
> can you build a kernel with USB_DEBUG, UHUB_DEBUG, XHCI_DEBUG, and
> EHCI_DEBUG debug options enabled, and with set these sysctl values
> in /etc sysctl.conf:
>
>    hw.usb.debug=1
>    hw.uhub.debug=1
>    hw.xhci.debug=1
>    hw.ehci.debug=1
>
> and then when it crashes, you can either collect a crash dump or if
> the serial console works properly, simply run this:
>
>   db{1}> show kernhist usbhist
>
> which will output a bunch debugging events from before the crash.
>
> thanks
>
>
> .mrg.
>