Re: hard hangs sometime after 22nd April and mutex problems

[Andrew Doran <ad%netbsd.org@localhost>]

On Mon, Apr 28, 2008 at 11:27:38AM +1000, Geoff Wing wrote:

> On Sunday 2008-04-27 15:28 +0100, Andrew Doran output:
> :On Sun, Apr 27, 2008 at 11:21:32AM +1000, Geoff Wing wrote:
> :> On i386 SMP I've been getting hard hangs - no response, pings, can't break
> :> into ddb - since sometime after the 22nd April.
> :Can you try running with LOCKDEBUG+DIAGNOSTIC please?
> 
> Righteo,
> it now seems to always die before the rc scripts are finished though at
> different times.  One trace had more problems after hitting panic/cpu_reboot
> and went through a sequence which panic'd/cpu_reboot'd again (#0/1 & #15/16)
> but another didn't.
> 
> Regards,
> Geoff
> 
> [160 column output follows]
> 
> #0  0xc036c553 in cpu_reboot (howto=260, bootstr=0x0) at 
> ../../../../arch/i386/i386/machdep.c:881
> #1  0xc02dfaf2 in panic (fmt=0xc05073cb "LOCKDEBUG") at 
> ../../../../kern/subr_prf.c:260
> #2  0xc02d8cea in lockdebug_abort1 (ld=0xcdd95e80, lk=0xc05df740, 
> func=0xc04639a7 "mutex_vector_enter",
>     msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == 
> curthread", dopanic=true) at ../../../../kern/subr_lockdebug.c:802
> #3  0xc02d9773 in lockdebug_abort (lock=0xcdde37cc, ops=0xc05594d0, 
> func=0xc04639a7 "mutex_vector_enter",
>     msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == 
> curthread") at ../../../../kern/subr_lockdebug.c:846
> #4  0xc02b4be3 in mutex_abort (mtx=0x0, func=0xc04639a7 "mutex_vector_enter", 
> msg=0xc05033c8 "assertion failed: MUTEX_OWNER(mtx->mtx_owner) == curthread")
>     at ../../../../kern/kern_mutex.c:325
> #5  0xc02b54a7 in mutex_vector_enter (mtx=0xcdde37cc) at 
> ../../../../kern/kern_mutex.c:712
> #6  0xc0326c3b in genfs_do_putpages (vp=0xcdde37cc, startoff=0, 
> endoff=9223372036854771712, origflags=<value optimized out>, busypg=0x0)
>     at ../../../../miscfs/genfs/genfs_io.c:935
> #7  0xc03271e8 in genfs_putpages (v=0xce44d84c) at 
> ../../../../miscfs/genfs/genfs_io.c:744
> #8  0xc03231b8 in VOP_PUTPAGES (vp=0xcdde37cc, offlo=0, offhi=0, flags=17) at 
> ../../../../kern/vnode_if.c:1763
> #9  0xc0230f57 in ffs_full_fsync (vp=0xcdde37cc, flags=0) at 
> ../../../../ufs/ffs/ffs_vnops.c:350
> #10 0xc0231071 in ffs_fsync (v=0xce44d978) at 
> ../../../../ufs/ffs/ffs_vnops.c:258
> #11 0xc0323bad in VOP_FSYNC (vp=0xcdde37cc, cred=0xcc065f00, flags=0, 
> offlo=0, offhi=0) at ../../../../kern/vnode_if.c:804
> #12 0xc022e52e in ffs_sync (mp=0xcdd33000, waitfor=2, cred=0xcc065f00) at 
> ../../../../ufs/ffs/ffs_vfsops.c:1397
> #13 0xc031992e in VFS_SYNC (mp=0xcdd33000, a=2, b=0xcc065f00) at 
> ../../../../kern/vfs_subr2.c:1102
> #14 0xc031ef78 in sys_sync (l=0xce4016e0, v=0x0, retval=0x0) at 
> ../../../../kern/vfs_syscalls.c:783
> #15 0xc0317e97 in vfs_shutdown () at ../../../../kern/vfs_subr.c:1810
> #16 0xc036c5d8 in cpu_reboot (howto=256, bootstr=0x0) at 
> ../../../../arch/i386/i386/machdep.c:867

The above is a red herring caused by the system getting confused during
panic.

> #17 0xc02dfaf2 in panic (fmt=0xc054bbec "kernel %sassertion \"%s\" failed: 
> file \"%s\", line %d") at ../../../../kern/subr_prf.c:260
> #18 0xc044471e in __kernassert (t=0xc04e3258 "diagnostic ", f=0xc050afa4 
> "../../../../kern/uipc_socket.c", l=599, e=0xc04e7526 "solocked(so)")
>     at ../../../../../../lib/libkern/__assert.c:50
> #19 0xc0305b8b in sofree (so=0xc2c0b5d4) at ../../../../kern/uipc_socket.c:599
> #20 0xc0305c6f in soabort (so=0xc2c0b5d4) at 
> ../../../../kern/uipc_socket.c:721
> #21 0xc0305d17 in soclose (so=0xc2c0b94c) at 
> ../../../../kern/uipc_socket.c:661
> #22 0xc02eff5c in soo_close (fp=0xce3425c0) at 
> ../../../../kern/sys_socket.c:255
> #23 0xc02a3686 in closef (fp=0xce3425c0) at 
> ../../../../kern/kern_descrip.c:757
> #24 0xc02a382f in fd_close (fd=3) at ../../../../kern/kern_descrip.c:648
> #25 0xc02a3a1e in fd_free () at ../../../../kern/kern_descrip.c:1491
> #26 0xc02a9fe1 in exit1 (l=0xce4016e0, rv=256) at 
> ../../../../kern/kern_exit.c:279
> #27 0xc02aa7d8 in sys_exit (l=0xce4016e0, uap=0xce44dd00, retval=0xce44dd28) 
> at ../../../../kern/kern_exit.c:183
> #28 0xc036fcb2 in syscall (frame=0xce44dd48) at 
> ../../../../arch/i386/i386/syscall.c:109
> #29 0xc01004fd in syscall1 ()

I think this one should be fixed... Could you please try with the latest
sources?

Andrew