port-amd64/55941: NetBSD 9+ /boot does not load, NetBSD 8+ /boot works ok - after update from 8-->9

[yancm%sdf.org@localhost]

>Number:         55941
>Category:       port-amd64
>Synopsis:       NetBSD 9+ /boot does not load, NetBSD 8+ /boot works ok - after update from 8-->9
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          support
>Submitter-Id:   net
>Arrival-Date:   Wed Jan 20 19:00:00 +0000 2021
>Originator:     Gene McNay
>Release:        9.1_Stable
>Organization:
>Environment:
NetBSD clarity.r-o-o-t.net 9.1_STABLE NetBSD 9.1_STABLE (GENERIC9.ipf) #0: Sat Jan 16 13:58:11 EST 2021  root%clarity.r-o-o-t.net@localhost:/usr/obj/sys/arch/amd64/compile/GENERIC9.ipf amd64

>Description:
I've been discussing my issue on netbsd-users here:
http://mail-index.netbsd.org/netbsd-users/2021/01/18/msg026437.html

amd64, Dell Precision WorkStation T3500, raidframe active

upgraded my home sever/gateway/firewall/nat from 8_Stable to 
9_Stable including adding ipf support to the kernel... everything 
seems to work except that the secondary bootloader /boot 
generated for n9 (both my self-compiled and binaries from 
the 9.1 release) are not being run from the first stage 
bootloader. I have verified that the new 1st stage bootloader 
is where the system hangs. If I switch /boot back to a binary 
from 8.1/8_Stable all is well, but the new /boot from 9+ just 
hangs.

Here are some details from a recent post to the thread above:
Below are a few other diagnostics I have tried in case they help
anyone else understand my issue...

# installboot -v -n -e /dev/wd0a
File system:         /dev/rwd0a
Boot options:        timeout 5, flags 0, speed 9600, ioaddr 0, console pc
# installboot -v -n -e /dev/rwd0a
File system:         /dev/rwd0a
Boot options:        timeout 5, flags 0, speed 9600, ioaddr 0, console pc
# installboot -v -n -e /dev/rwd0
File system:         /dev/rwd0
installboot: Invalid magic in existing bootstrap
installboot: Edit bootstrap operation failed
# installboot -v -n -e /dev/wd0
File system:         /dev/rwd0
installboot: Invalid magic in existing bootstrap
installboot: Edit bootstrap operation failed
# installboot -v -n -e /dev/wd0a
File system:         /dev/rwd0a
Boot options:        timeout 5, flags 0, speed 9600, ioaddr 0, console pc
# installboot -v -n -e /dev/wd1a
File system:         /dev/rwd1a
Boot options:        timeout 5, flags 0, speed 9600, ioaddr 0, console pc


[NOTE: this was the most recent step I took to alter the 
1st stage bootloader to print a unique string just so I could 
be sure this was in fact the new 1st stage bootloader that 
was running]
# installboot -t raid -v -o timeout=8 /dev/rwd0a \
/usr/obj/destdir.amd64/usr/mdec/bootxx_ffsv1
File system:         /dev/rwd0a
File system type:    raid (blocksize 16384, needswap 0)
Primary bootstrap:   /usr/obj/destdir.amd64/usr/mdec/bootxx_ffsv1
Boot options:        timeout 8, flags 0, speed 9600, ioaddr 0, console pc
# installboot -v -n -e /dev/wd0a
File system:         /dev/rwd0a
Boot options:        timeout 8, flags 0, speed 9600, ioaddr 0, console pc
# installboot -t raid -v -o timeout=9 /dev/rwd1a
/usr/obj/destdir.amd64/usr/mdec/bootxx_ffsv1
File system:         /dev/rwd1a
File system type:    raid (blocksize 16384, needswap 0)
Primary bootstrap:   /usr/obj/destdir.amd64/usr/mdec/bootxx_ffsv1
Boot options:        timeout 9, flags 0, speed 9600, ioaddr 0, console pc
# installboot -v -n -e /dev/wd1a
File system:         /dev/rwd1a
Boot options:        timeout 9, flags 0, speed 9600, ioaddr 0, console pc

# /usr/sbin/dumpfs -s /dev/rwd1a
dumpfs: /dev/rwd1a: could not find superblock, skipped
# /usr/sbin/dumpfs -s /dev/wd1a
dumpfs: /dev/wd1a: Device busy
# /usr/sbin/dumpfs -s /dev/raid0a
dumpfs: /dev/raid0a: Device busy
# /usr/sbin/dumpfs -s /dev/rraid0a
file system: /dev/rraid0a
format  FFSv1
endian  little-endian
magic   11954           time    Tue Jan 19 14:18:06 2021
superblock location     8192    id      [ 5e039dba 70e4d4fd ]
cylgrp  dynamic inodes  4.4BSD  sblock  FFSv2   fslevel 4
nbfree  34132   ndir    2683    nifree  117464  nffree  345
ncg     6       size    524288  blocks  516119
bsize   16384   shift   14      mask    0xffffc000
fsize   2048    shift   11      mask    0xfffff800
frag    8       shift   3       fsbtodb 2
bpg     10923   fpg     87384   ipg     21504
minfree 5%      optim   time    maxcontig 4     maxbpg  4096
symlinklen 60   contigsumsize 4
maxfilesize 0x000400400402ffff
nindir  4096    inopb   128
avgfilesize 16384       avgfpdir 64
sblkno  8       cblkno  16      iblkno  24      dblkno  1368
sbsize  2048    cgsize  16384
csaddr  1368    cssize  2048
cgrotor 0       fmod    0       ronly   0       clean   0x02
wapbl version 0x1       location 2      flags 0x0
wapbl loc0 1209088      loc1 2048       loc2 512        loc3 125
usrquota 0      grpquota 0
flags   wapbl
fsmnt   /
volname         swuid   0

# /usr/bin/file -s /dev/rwd0a
/dev/rwd0a: DOS/MBR boot sector, BSD disklabel
# /usr/bin/file -s /dev/wd0a
/dev/wd0a: writable, no read permission
# /usr/bin/file -s /dev/wd0
/dev/wd0: DOS/MBR boot sector; partition 1 : ID=0xa9, active, start-CHS
(0x0,32,33), end-

# raidctl -s raid0
Components:
           /dev/wd1a: optimal
           /dev/wd0a: optimal
No spares.
Component label for /dev/wd1a:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 20191223, Mod Counter: 808465757
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 312574848
   RAID Level: 1
   Autoconfig: Yes
   Root partition: Soft
   Last configured as: raid0
Component label for /dev/wd0a:
   Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 20191223, Mod Counter: 808465757
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 312574848
   RAID Level: 1
   Autoconfig: Yes
   Root partition: Soft
   Last configured as: raid0
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
# mount
/dev/raid0a on / type ffs (log, local)
/dev/raid0f on /var type ffs (log, local)
/dev/raid0e on /usr type ffs (log, local)
/dev/raid0g on /home type ffs (log, local)
/dev/raid0k on /sambashare type ffs (log, NFS exported, local)
/dev/raid0h on /usr/src type ffs (log, local)
/dev/raid0i on /usr/pkgsrc type ffs (log, local)
/dev/raid0j on /usr/obj type ffs (log, local)

>How-To-Repeat:
AFAIK, this only happens on my machine or others would be reporting it.

On my machine, all I need to do to turn it on is put a 9+ /boot in and turn it off with an 8+ /boot.
>Fix:
Only fix is to drop back to a /boot from 8+ and/or boot from a CD and use the 2nd stage bootloader from that to invoke my kernel