Re: NetBSD netboot with maximal cross-architecture sharing

[Jason Thorpe <thorpej%me.com@localhost>]

]
> On May 12, 2020, at 1:10 PM, Chris Hanson <cmhanson%eschatologist.net@localhost> wrote:
> 
> On May 10, 2020, at 4:55 PM, Jason Thorpe <thorpej%me.com@localhost> wrote:
>> 
>> On May 10, 2020, at 1:58 PM, Chris Hanson <cmhanson%eschatologist.net@localhost> wrote:
>>> 
>>> Are there any documented best practices for setting NetBSD up in this way? If I want to set up (say) NetBSD-current to netboot on several older systems with different architectures, I could naïvely just create a distinct root for each system, but I’d really like to share what I can (beyond home directories) since so much of the OS content will be the same.
>> 
>> /usr/share is named as such precisely because it's sharable across all platforms.
> 
> Sure, but is there *more* that can be shared across platforms?

Oh, **totally**.

> Taking an example from my own network, I have both sun3 and hp300 hardware on which I can network-boot NetBSD. Is there a good way to set up sharing most of the m68k-architecture binaries while not sharing the sun3-platform and hp300-platform binaries?

So, for the most part, there should not be separate "sun3 platform" and "hp300 platform" binaries... This has been a moderate issue for years, but no one has really put forth the effort to address it.

So, assuming default m68k ELF build options (which I believe means "hard float everywhere"):

/bin -> same everywhere
/lib -> same everywhere
/libexec -> same everywhere
/sbin -> same everywhere except for /sbin/disklabel which has a very tangled history and needs to die in a fire.
/usr/bin -> same everywhere
/usr/lib -> same everywhere except maybe libkvm but I think that does run-time checks
/usr/libexec -> same everywhere
/usr/sbin -> mostly the same, except:
-> /usr/sbin/crash -- maybe differences handling sun3 vs Motorola MMU in crash dumps?  Libkvm maybe hides those for us, I can't quite remember.
-> /usr/sbin/eeprom -- built on sun3, not on hp300 (it's modeled after the SunOS eeprom program).
-> /usr/sbin/installboot -- might be OK?
-> /usr/bin/sysinst -- tons of platform-specific stuff in there.

I might be missing a few things, but as you can see, there is a ton of sharable content.


> 
> I expect that with a bit of work I’ll be able to set up the following for netboot for a particular version of NetBSD:
> 
> - /home to be fully shared among all my systems running NetBSD
> - /usr/share to be fully shared among all my systems running a particular version of NetBSD
> - / to be entirely distinct per-system, mainly as a host for symbolic links, /etc content, /var content, and /tmp
> - most of /bin, /sbin, /lib, /usr to be shared per-architecture (m68k, ppc, mips, etc.)
> - kernel and modules to be shared per-platform (sun3, hp300, pmax, etc.) and not per-architecture
> 
> Am I on the right track here? Or does the fact that each platform is basically a world build make it risky to share per-architecture binaries too? And will the structure of the archives produced by a NetBSD build right now make doing the above difficult, enough so that I’d be better off just burning disk space on an install per system, with just shared /home and /usr/share?

It's pretty close!  But yah, there are a few annoying platform-specific things squirreled away in otherwise fully-sharable directories.

> 
>  -- Chris
> 
> 
> 
> 

-- thorpej