Re: kern/40474: Kernel panic after remounting raid root with softdep

[David Holland <dholland-bugs%netbsd.org@localhost>]

On Sun, Jan 25, 2009 at 11:05:00AM +0000, kivinen%iki.fi@localhost wrote:
 >      The system has mirrored raid as root, and when I boot it up
 >      and then run mount -o softdep -u / the command works, but after
 >      that next time I start "vi" the machine panics. The mount
 >      command does not show that the softdep took effect (i.e
 >      it still list options as local), and if I can still 
 >      run some other programs from the root filesystem. The 
 >      file edited with vi does not matter, and crash happens
 >      even if no file is given. If this is tried on the second 
 >      (non root disk) then this does not happen, even if the 
 >      vi binary is copied to the other disk (i.e. mount the
 >      another disk without softdep, remount it with -u -o softdep
 >      and then try to run vi from there do work). 
 > 
 >      I assume this might be related to somehow to the way 
 >      vi creates temp files or something.

I'm pretty sure turning softdep on after mount is a known problem, and
that there's an existing PR about it, but I can't find one. Anyway, I
doubt it's got much to do with either vi or raid.

I think I know what's going on, maybe, but I don't have time right now
to look into it in any detail, and unfortunately I don't think anyone
else is interested in fixing softdep issues... :(

The immediate problem appears to be that it's allocating the same
inode twice. I think this is probably happening either because of a
general known problem with mount updates not flushing things properly
(e.g. PR 30525)... or because softdep uses different rules for
handling free object bitmaps from baseline ffs, and after the mode
transition not all filesystem state is arranged the way softdep
expects.

Did the fsck after the crash show any problems? And, if you reboot
again without having used softdep, boot to single user, and force
running a fsck, does *that* show any problems? (In fact, it's probably
a good idea to do such a fsck on general principles; fsck behaves
differently if it thinks softdep was in use, so crashing in this
fashion might have confused it.)


If you have a test machine to crash on, the following information
might be useful, maybe:

   - If you sync after the remount, does the crash still occur?
     (I expect it will, but it would be interesting if it didn't.)

   - Can you reproduce the crash without vi but instead by manually
     creating a scratch file in (probably) /var/tmp or
     /var/tmp/vi.recover?

   - If you boot single-user, remount / read-write, sync, remount /
     again to turn on softdep, sync, and then run vi, does it crash?
     If not, does omitting any of the syncs change the behavior?

   - If not, and you do the same, but create a scratch file in /tmp or
     /var/tmp before turning on softdep, does it then crash? If not,
     does creating a lot of scratch files instead of just one trigger
     the crash?

   - It might also be interesting to deliberately hit the power switch
     at various points, because this allows comparing what's been
     written to disk with what's being kept in memory, and any
     discrepancy is interesting. However, this requires mucking about
     with fsdb and I'm not sure exactly what to look for either.

(None of this is worth crashing anything other than a test machine to
find out, though.)

-- 
David A. Holland
dholland%netbsd.org@localhost