misc/41594: record rc messages

[raeburn%raeburn.org@localhost]

>Number:         41594
>Category:       misc
>Synopsis:       record rc messages
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    misc-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Sun Jun 14 10:55:00 +0000 2009
>Originator:     Ken Raeburn
>Release:        NetBSD 5.0
>Organization:

>Environment:
System: NetBSD raeburn.org 5.0 NetBSD 5.0 (GENERIC) #0: Sun Apr 26 18:50:08 UTC 
2009 
builds%b6.netbsd.org@localhost:/home/builds/ab/netbsd-5-0-RELEASE/i386/200904260229Z-obj/home/builds/ab/netbsd-5-0-RELEASE/src/sys/arch/i386/compile/GENERIC
 i386
Architecture: i386
Machine: i386
>Description:

Messages displayed during the rc part of the boot process scroll away
too quickly.  If X is run on the console, they'll go away altogether.
Or if you have a system with multiple card media slots that show up as
sd devices and you run smartd and let it check all devices, the kernel
may print a whole screenful of "medium not present" messages, quickly
hiding anything that came before.

The messages can be important.  For example, the first time I booted
netbsd 5 after upgrading, I saw something go by about ipnat, but
didn't catch it.  Whatever it was, it might've helped to explain why
the NAT configuration completely failed to take effect (RFC 1918
addresses showed up on outgoing traffic) despite showing up in "ipnat
-l" listing and despite repeated attempts to use the rc scripts to
turn ipfilter and ipnat off and back on.  I didn't see any such
message on later reboots, nor did the problem show up again.  But
*something* broke that first time, and I have very little data on what
it might have been.

Kernel messages are stored in dmesg.boot, but user-mode messages
generated by scripts just go away.  They should be available someplace.

>How-To-Repeat:
        reboot, then try to find the messages that have scrolled away
>Fix:

One possible approach: I don't recall if NetBSD has it, but some
systems have an ioctl to let a process grab console output.  Run such
a program for the duration of /etc/rc, writing the data to a file in
/var/run -- but if the file system is read-only, buffer the output up
to some reasonable limit, and try re-opening the file on receipt of a
signal, sent by an rc script.  If re-opening succeeds, write out the
buffered output.

I think the ioctl usually redirects console output completely, so it
never shows up on the actual console; that would be inappropriate
here.  We'd want something that grabs a copy of the console output,
and lets a copy also get displayed.

Another option might be to, instead of writing out a special file,
syslog the messages after syslogd starts up.  (Or do both.)  Then the
option could be given of continuing to do such logging even after the
rc script finishes (e.g., if text-mode console logins aren't expected,
or have a way to disable the logging at login time), and an admin can
watch console messages remotely via a log file.

Mac OS X and one or two others I don't specifically remember have a
useful variation, which is that all daemon processes get stdout/stderr
fed to console-log files but with process identification attached, a
la "logger".  That would probably need to be done with an excessive
number of logger processes (though they could go away if stdout and
stderr are closed), or some master controller process like launchd.