current issues on Octeon (ER-4)

[Denis Ovsienko <denis%ovsienko.info@localhost>]

Hello list.

I've been running NetBSD on EdgeRouter-4 for a while using various HEAD
snapshots, so far with your help this unusual host managed to do around
1500 builds for ci.tcpdump.org and I would like to keep it doing this
job.  Hopefully this feedback will help it happen.

This March I saw the announce of NetBSD 10 beta and tried replacing a
rather old HEAD snapshot from 202207052210 with a snapshot of
netbsd-10.  That immediately failed due to a kernel startup problem (PR
57280), which now has a bug fix in HEAD, but still not in netbsd-10.  It
would be nice to backport the bug fix.

Meanwhile I tried HEAD again, which worked a bit better, but not
without problems.  First of all, octeon.img.gz still has the problem
that (I think) we discussed much earlier on this list: when the kernel
boots, it looks for the root filesystem on sd0a, but in this disk image
the root filesystem appears to be on dk1:

[   1.8199870] sd0 at scsibus0 target 0 lun 0: <SanDisk, SDDR-B531,
2920> disk removable
[   1.8307603] sd0: fabricating a geometry
[   1.8307603] sd0: 116 GB, 118900 cyl, 64 head, 32 sec, 512 bytes/sect
x 243507200 sectors
[   1.8436010] sd0: fabricating a geometry
[   1.8499885] dk0 at sd0: "octeon-boot", 163840 blocks at 32768, type:
ntfs
[   1.8499885] dk1 at sd0: "octeon-root", 794624 blocks at 196608,
type: ffs
[   1.8644615] swwdog0: software watchdog initialized
[   1.8644615] WARNING: 4 errors while detecting hardware; check system
log.
[   1.8763753] boot device: sd0
[   1.8799893] root on sd0a dumps on sd0b
[   1.8799893] vfs_mountroot: can't open root device
[   1.8799893] cannot mount root, error = 16
[   1.8933204] root device (default sd0a): 

If the user enters "dk1" on the console, the system boots:

[   1.8933204] root device (default sd0a): dk1
[   3.2230885] dump device: 
[   3.2230885] file system (default generic): 
[   3.3075380] root on dk1
[   3.3075380] root file system type: ffs
[   3.3191181] kern.module.path=/stand/evbmips/10.99.4/modules
[   3.3191181] WARNING: no TOD clock present
[   3.3297187] WARNING: using filesystem time
[   3.3337952] WARNING: CHECK AND RESET THE DATE!
[   3.3382237] init path (default /sbin/init): 
[   4.3124565] init: trying /sbin/init
Sun May 14 08:43:13 UTC 2023
Starting root file system check:
/dev/rdk1: file system is clean; not checking
gpt: /dev/rsd0: No valid PMBR partition found
[   4.7024522] dk0 at sd0 (octeon-boot) deleted
[   4.7124566] dk0 at sd0: "octeon-boot", 163840 blocks at 32768, type:
ntfs
[   4.7124566] dk1 at sd0: "octeon-root", 243302400 blocks at
196608, type: ffs /dev/rsd0: Partition 2 resized: 196608 243302400
Resizing / (NAME=octeon-root)

This way, with the disk image unattended reboots still do not work,
which is a bit inconvenient.  It would be nice to have a fix for this
bug if possible.

The next thing I tried was booting netbsd-INSTALL_OCTEON and going
through the manual menu-based installation the same way as many times
before (MBR partition table with a 100MB FAT partition for U-boot needs
and a NetBSD partition for the disklabel).  As it turned out, this
method still experiences an old problem: after the installation the MBR
partition table is invalid. However, the two partitions are where they
are supposed to be and with the right data, so after a quick manual
one-off MBR repair the block device is good.  That's another bug that
it would be better not to have if possible.

Using this bootable snapshot of HEAD, the host completed a scripted
build of its usual packages from pkgsrc-2023Q1, which took several
hours and went almost perfect except it did hang once in the middle of
process for reasons unknown (neither serial console nor SSH were
attached at the time, so there was no diagnostics).  Anyway, it was
good enough to compile python310, perl and ruby31-base among other
things, so I thought overall it was going to be the next working
snapshot.

However, before long another problem emerged:

netbsd-mips64# git --version
[1]   Abort trap              git --version

I tried to get a backtrace of this, but gdb did not recognize the git
binary as something it knows how to run.  I wonder if this problem
reproduces for anybody else.  The binary is from the pkgsrc package
git-base-2.40.0, which builds and works fine on NetBSD 9.3/AArch64.

As before, this ER-4 host is available for remote access and debugging
on request.

On a related note, if any NetBSD developer would like to have a small
Octeon host for tests or whatever, I have a spare ERLite-3 remaining
from my earlier experiments.  This is a modified device: it has a
rather oversized, but slow and silent fan and its internal USB 2.0 port
is connected to a socket on the back of the device, so trying different
disk images is quick and easy:

https://ovsienko.info/photo/picture.php?/2819/category/7

If you think this device can help your work on NetBSD, let me know and
you can have it for free.

-- 
    Denis Ovsienko