strange behavior of syslogd

[David Holland <dholland-tech%netbsd.org@localhost>]

On some of my production machines I have syslogd configured to send
log messages to me on the tty where I'm logged in, by username.

Last Thursday afternoon I rebooted one of those machines and then went
home for the long weekend. Today I discovered that since the reboot,
log messages were going to disk but not to the tty. More curiously
still, logging in a second time (on ttyp1 as well as ttyp0) caused log
messages to start going to both ttys... and logging the second session
off reverted to the previous behavior of getting none.

This machine is running current from 20080531, and over the preceding
month running the same code I hadn't seen this problem. (Nor have I
seen anything like it before that I can recall.)

Clearly there's something wrong with the utmp handling, but so far I
can't find anything that explains the behavior. This afternoon I
thought the way syslogd was calling getutentries() and freeutentries()
was corrupting the malloc arena, but closer inspection shows that it
shouldn't be. Instead, it looks like it forces the utmpentry code to
reread utmp for every syslog message; that's a bug, but doesn't
explain the behavior I was seeing. (If anything, it makes it more
mysterious, because it means it reread all the utmp entries for each
log message and each time managed to lose or skip over the one it
needed to find.)

I need to dig around some more (and unfortunately, because I thought I
understood what was wrong, I went and restarted syslogd and probably
won't be able to duplicate the problem again) but it is very strange.

Anyone have any bright ideas?

-- 
David A. Holland
dholland%netbsd.org@localhost