RAIDframe boot issues on amd64 (BIOS boot)

[Matthias Petermann <mp%petermann-it.de@localhost>]

Hello all,

I have a non-UEFI PC with BIOS boot. The PC has two hard disks of the 
same size that I use to configure a RAIDframe RAID 1 on which the root 
filesystem lives and where I want to boot from. Depending on the start 
sector of the partitions that form the RAID components, there is a 
different behavior at boot for which I am looking for the cause.

## Preconditions / Setup

The system I use is NetBSD/amd64 9.2_STABLE, built from the sources 
around 2021-09-14.

The creation and partitioning of RAID 1 via sysinst works largely well 
menu-driven. I have tried various combinations of partitioning schemes 
and have determined the following to be the most convenient for my purpose:

1) Partitioning the raw disks wd0/wd1 via GPT partition table (a single 
RAID partition on each - wedge dk0/dk1).
2) Creation of the RAID from wedges dk0/dk1 as components
3) partitioning of RAID by NetBSD disklabel (raid0a is root, raid0b swap 
and so on)

These three steps, as well as the installation of the sets / 
configuration of network, user account and services to be started at 
boot time can be done completely in sysinst. There are only a few small 
things to refine in the shell after the installation is complete:

1) Set the GPT partitions to active and write the GPT-MBR boot code to 
sector 0.

```
gpt biosboot -A -i 1 wd0
gpt biosboot -A -i 1 wd1
```

2) write the primary bootstrap code PBR to the beginning of the RAID 
partitions (writing via sysinst should be skipped, because sysinst tries 
to write to raid0a which is not correct according to wiki[1])

```
installboot -o timeout=30 -v /dev/rdk0 /usr/mdec/bootxx_ffsv2
installboot -o timeout=30 -v /dev/rdk1 /usr/mdec/bootxx_ffsv2
```

3) set raid0 to forceroot

```
raidctl -A forceroot raid0
```

## Observations

There are two cases depending on the start sector of the RAID wedges

a) if wedges dk0/dk1 each start in sector 63

 - the system boots without problems.

b) if wedges dk0/dk1 each start in sector 2048

 - the following output appears:

```
NetBSD/x86 ffsv2 Primary Bootstrap

@@@@@@@@@
>> NetBSD/x86 BIOS Boot, Revision 5.11 ....
>> Memory: 610/.......
Press return to boot.....
booting hd0a:netbsd - starting in....
open netbsd: input/output error
```

 - the @'s appear one by one ~ every second. While this process, the 
drive wd0 LED flashes so it looks it is seeking something

 - finally it seems to find the (secondary?) boot program, but this 
doesn't seem to find raid0a

 - anyway, in this stage I can enter the boot prompt. From there, I can 
get a successful boot by entering "raid0a:netbsd"

## Further Detail

Interesting fact: I did the identical installation in a VirtualBox. 
Here, both cases a) and b) work flawlessly.

## Questions

 - does anyone know what the output of @'s means? I could not derive 
this from the source code of the bootxx_ffsv2 unfortunately, also did 
not find something in the error code explainations in the wiki[2]

 - could it be a BIOS limitation of my specific model and if so - how 
to explain it?

 - if the workaround is to have the wedges dk0/dk1 start in sector 63 - 
what effect does this have on the alignment? (I had read that an offset 
of 2048 is favorable).


Many thanks in advance & kind regards
Matthias


[1] https://www.netbsd.org/docs/guide/en/chap-rf.html#chap-rf-moving-files

[2] https://wiki.netbsd.org/tutorials/how_netbsd_boots_on_x86/