Modularize sun2 kernel (Re: CVS commit: src/sys/arch/sun2/conf)

[Rin Okuyama <rokuyama.rk%gmail.com@localhost>]

(added port-sun2 and thorpej)

On 2020/03/08 17:40, matthew green wrote:
> "Rin Okuyama" writes:
> > Module Name:	src
> > Committed By:	rin
> > Date:		Sun Mar  8 06:25:10 UTC 2020
> >
> > Modified Files:
> > 	src/sys/arch/sun2/conf: GENERIC
> >
> > Log Message:
> > Retire md(4) in favor of tmpfs provided by module,
> > though both are not useful for 8MB RAM system...

...

> it's cool you got modules on sun2 working... however, the
> module version of a thing is going to use more real memory
> than the built-in version, so i'm not sure where this is
> going yet...

Sorry for the late reply. I was confused by weird kernel behaviors when
memory shortage. But, I think that I slowly came to understand that :).

Yes, modular kernel requires more memory, mainly because KERN_AS is
changed from ``library'' to ``obj'':

    https://nxr.netbsd.org/xref/src/sys/lib/libkern/Makefile.inc#8

This pulls in all libkern routines to kernel.

However, I think that modularization is the only way to prolong sun2
port as far as I can tell.

We need to reduce kernel text as much as possible. This is because

(1) Our bootloader mapps only 2MB for kernel image.
(2) Even though bootloader can load kernel, it hangs here and there,
    if kernel size is near 2MB.

For (1), our GENERIC is very close to this limit, even though almost
all additional features are stripped off. I managed to map more 128KB
memory, (maybe) not used by firmware, at least on TME:

    http://www.netbsd.org/~rin/sun2_bootloader_20200315.patch

But I don't know whether it works or not on real hardwares.

(2) is more serious. When kernel hangs, it is trapped in pagedaemon.
I built a kernel with DIAGNOSTIC and LOCKDEBUG, but no failures are
detected. Now, I imagine its scenario as follows:

Our kernel allocates many memory on demand. Hangs may be the normal
operation (not a bug) when memory is allocated with KM_SLEEP or
M_WAITOK, but pagedaemon cannot free memory. Swap is not ready in
early boot stage, or all memory is wired, or thrashing takes place
like ping-pong; suppose two memory segments are required at the same
time. pagedaemon swaps out segment 1 to swap in segment 2, and
immediately after, it frees segment 2 to pull in segment 1.

Then, ``deadlock'' is effectively formed, which is cannot detected by
neither DIAGNOSTIC nor LOCKDEBUG.

Hangs due to (2) are remarkably mitigated by modularization. More
features can be loaded by modload(8), even if kernel stops working
when that are statically linked into kernel. This is because kernel
text is wired, whereas modules are pageable.

> 8MB?  luxury?  :)  i've been testing in 4MB tme :)

Sorry, s/8MB/7MB/ in the commit log. Framebuffer and etc. reside at
PA 0x700000 and above.

I think that we can no longer support 4MB system because of (2); hangs
due to (2) are much more serious for 4MB system than it is in 7MB system.
Modern kernel allocates too much things on demand rather than statically
allocating them in data segment...

GENERIC became not working again due to (2) (even for 7MB). Can I go
further, i.e., strip more features that can be provided by modules, or
do you have any other ideas?

P.S.
In many cases of hangs due to (2), I can break into DDB from console.
However, in some cases, especially due to "ifconfig lo0 127.0.0.1" in
/etc/rc, it does not even accept input from console. I suspect that
someone in our network code allocates memory with KM_SLEEP or M_WAITOK
in the interrupt context, or with spl raised...

Thanks,
rin