Re: mutex lock panic on halt/shutdown unmounting GPT/RAID filesystem

[David Young <dyoung%pobox.com@localhost>]

On Tue, Jul 05, 2011 at 01:27:49PM -0500, John D. Baker wrote:
> On Tue, 5 Jul 2011, Greg Oster wrote:
> 
> >It may not be related, but I suspect you may be the first person to be
> >testing the RF_RAID5_RS bits :) It wouldn't surprise me if there were a
> >few wrinkles with that code and NetBSD.
> 
> I've used it for years.  On my previous file server, I actually got it to
> work once (as in failed component replaced and reconstructed in place).
> The other times, a second unit failed during reconstruction but the
> raid continued to run OK in degraded mode.
> 
> The current file-server is built with it, but hasn't been put to the
> test (no failed components since being placed into service in 2009).
> The current fileserver is running NetBSD-4.0_STABLE.  It has never
> lost the RAID, but its system disk keeps eating itself and causing
> panics (mostly freeing free block and mangled directory entry).
> Also, it takes 17+ hours to check/reinitialize parity, so that's why I'm
> building another one on hardware that likes the netbsd-5 branch.
> 
> 
> In response to David Young's posting:
> 
> Jul  5 13:21:25 yggdrasil shutdown: reboot by sysop: Jul  5 13:21:33
> yggdrasil syslogd: Exiting on signal 15
> syncing disks... 7 done
> unmounting file systems...Mutex error: mutex_vector_enter: locking against 
> myself
> 
> lock address : 0x00000000c4d5a71c
> current cpu  :                  0
> current lwp  : 0x00000000cf9557c0
> owner field  : 0x00000000cf9557c0 wait/spin:                0/0
> 
> panic: lock error
> fatal breakpoint trap in supervisor mode
> trap type 1 code 0 eip c057dc0c cs 8 eflags 246 cr2 bbb9890c ilevel 0
> Stopped in pid 485.1 (reboot) at        netbsd:breakpoint+0x4:  popl    %ebp
> db{0}> bt 
> breakpoint(c0a89ee6,cfa097a8,c0ab9440,c04c7cbf,0,1,0,0,cfa097a8,cf9557c0)
> at net
> bsd:breakpoint+0x4 
> panic(c0a4c51d,c0a4a16f,c0852252,c0a4a13e,c4d5a71c,0,cf9557c0,cf418730,c4d5a71c,
> cf9557c0) at netbsd:panic+0x1b0 
> lockdebug_abort(c4d5a71c,c0ab62f0,c0852252,c0a4a13e,cf418730,1,cfa0987c,c049ce72
> ,c4d5a71c,c0852252) at netbsd:lockdebug_abort+0x2d 
> mutex_abort(c4d5a71c,c0852252,c0a4a13e,cf418730,a800,0,cfa0981c,c0514314,c0b7ac2
> 0,cdeb59c0) at netbsd:mutex_abort+0x2e
> mutex_vector_enter(c4d5a71c,0,0,4,7,c4bfe000,cfa098cc,cdeb5740,a8,a8)
> at netbsd:
> mutex_vector_enter+0x262 
> dkwedge_del(cfa098d0,cf35de48,10,306b64,c0b15b8c,cdeb5740,1,c081019d,cdeb5748,c4
> d6c488) at netbsd:dkwedge_del+0x188 
> dkwedge_delall(c4cb2828,c0aa3220,0,c0826120,1203,cf9557c0,cfa099fc,c04bdbc4,1203
> ,3) at netbsd:dkwedge_delall+0x61
> raidclose(1203,3,6000,cf9557c0,6000,3,6,3,cf6cc5c0,0) at
> netbsd:raidclose+0x12f
> bdev_close(1203,3,6000,cf9557c0,0,0,cfa09a4c,1203,6000,0) at 
> netbsd:bdev_close+0
> x84 
> spec_close(cfa09a58,20002,cfa09a6c,c0509038,cf6cc5c0,c08537a0,cf6cc5c0,3,fffffff
> f,3) at netbsd:spec_close+0x237 
> VOP_CLOSE(cf6cc5c0,3,ffffffff,c04f7335,c4d5a600,0,cfa09aac,c046a9e6,cf6cc5c0,3)
> a
> t netbsd:VOP_CLOSE+0x6c
> vn_close(cf6cc5c0,3,ffffffff,c0850ac0,a800,cf9557c0,cfa09adc,c04bdbc4,a800,3) 
> at
>  netbsd:vn_close+0x4e
> dkclose(a800,3,6000,cf9557c0,6000,3,6,3,cf4187e8,0) at
> netbsd:dkclose+0xc6
> bdev_close(a800,3,6000,cf9557c0,0,0,cfa09b2c,a800,6000,0) at 
> netbsd:bdev_close+0
> x84 
> spec_close(cfa09b38,20002,cfa09b4c,c0509038,cf4187e8,c08537a0,cf4187e8,3,fffffff
> f,c57ed000) at netbsd:spec_close+0x237 
> VOP_CLOSE(cf4187e8,3,ffffffff,c03b9066,0,c4d6c8cc,c4d6c880,cf701000,cf701000,cf7
> 01024) at netbsd:VOP_CLOSE+0x6c 
> ffs_unmount(cf701000,80000,cf8d63c0,0,cf701000,0,cfa09bcc,c050732f,cf701000,8000
> 0) at netbsd:ffs_unmount+0x1c9 
> VFS_UNMOUNT(cf701000,80000,cf8d63c0,0,2001018,cf6cc398,1,cf701000,cf700000,cf955
> 7c0) at netbsd:VFS_UNMOUNT+0x26
> dounmount(cf701000,80000,cf9557c0,0,cfa09c08,7,0,cf9557c0,cfa09d00,0)
> at netbsd:
> dounmount+0x13f 
> vfs_unmountall(cf9557c0,0,0,c048976d,cde8b630,0,cfa09c3c,c058456b,0,cf9557c0)
> at
>  netbsd:vfs_unmountall+0x63
> vfs_shutdown(0,cf9557c0,0,0,cfa09d00,0,cfa09cec,c04b83c4,0,0) at
> netbsd:vfs_shut
> down+0x8d cpu_reboot(0,0,0,0,0,0,c0a4b571,0,23,fffffffe) at
> netbsd:cpu_reboot+0x13b
> sys_reboot(cf9557c0,cfa09d00,cfa09d28,0,0,bfbfeed8,8049144,2,1,1) at 
> netbsd:sys_
> reboot+0x74 syscall(cfa09d48,b3,ab,1f,1f,1,d,bfbfeed8,0,256) at
> netbsd:syscall+0xbd
> db{0}>

Thanks for the backtrace.  Is it possible to try -current?  There have
been many changes in the raidframe/dkwedge area.  One of them has
probably fixed this bug.

Dave

-- 
David Young             OJC Technologies
dyoung%ojctech.com@localhost      Urbana, IL * (217) 344-0444 x24