kern/45827: panic: LOCKDEBUG (Reader / writer lock error: lockdebug_unlocked: no shared locks held by LWP)

["Greg A. Woods" <woods%planix.ca@localhost>]

>Number:         45827
>Category:       kern
>Synopsis:       panic: LOCKDEBUG (Reader / writer lock error: 
>lockdebug_unlocked: no shared locks held by LWP)
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Jan 12 23:15:00 +0000 2012
>Originator:     Greg A. Woods
>Release:        NetBSD 5.1_STABLE (2012/01/08)
>Organization:
Planix, Inc.; Kelowna, BC; Canada
>Environment:
System: NetBSD more 5.1_STABLE NetBSD 5.1_STABLE (GENERIC) #3: Sun Jan  8 
21:53:14 PST 2012  
woods@once:/rest/build/woods/once/netbsd-5-i386-i386-ppro-obj/rest/work/woods/m-NetBSD-5/sys/arch/i386/compile/GENERIC
 i386
Architecture: i386
Machine: i386
>Description:

        My shiny new-to-me Dell PE2950 (dual quad-core 32GB) system died
        with the following panic during its first real load testing:

        ("build.sh j 4" -- it was still in the middle of building tools)

Reader / writer lock error: lockdebug_unlocked: no shared locks held by LWP

lock address : 0x00000000c0d53cc0 type     :     sleep/adaptive
initialized  : 0x00000000c04e0c43
shared holds :                  0 exclusive:                  0
shares wanted:                  0 exclusive:                  0
current cpu  :                  3 last held:              65535
current lwp  : 0x00000000de918d20 last held: 000000000000000000
last locked  : 0x00000000c04e02e1 unlocked : 0x00000000c04e04d2
owner/count  : 000000000000000000 flags    : 0x0000000000000008

Turnstile chain at 0xc0d54200.
=> No active turnstile for this lock.

panic: LOCKDEBUG
Begin traceback...
WARNING: SPL NOT LOWERED ON SYSCALL EXIT
copyright(d6f23503,0,0,de721cf4,d6f23e40,d6f23dd8,c0d53cc0,c0c3c0d8,0,c04e02e1) 
at 0xc0b91150
Bad frame pointer: 0xd6f23540
End traceback...

dumping to dev 4,1 offset 96479359
dump 150 149 148 147 146 WARNING: SPL NOT LOWERED ON SYSCALL EXIT
145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 
125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 
105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 
81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 
55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 
WARNING: SPL NOT LOWERED ON SYSCALL EXIT
29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 
succeeded


[[....]]

Checking for core dump...
savecore: reboot after panic: panic: LOCKDEBUG
savecore: system went down at Thu Jan 12 12:08:59 2012
savecore: /var/crash/bounds: No such file or directory
savecore: writing compressed core to /var/crash/netbsd.0.core.gz
savecore: writing compressed kernel to /var/crash/netbsd.0.gz
savecore: (null): Bad address



# gdb ~woods/tmp/netbsd.gdb /var/crash/netbsd.0.core                            
                                                   
GNU gdb 6.5
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386--netbsdelf"...
"/var/crash/netbsd.0.core" is not a core dump: File format not recognized
(gdb) target kvm
#0  0xc04de9c9 in mi_switch (l=0xc0c3bd80) at 
/rest/work/woods/m-NetBSD-5/sys/kern/kern_synch.c:767
767     /rest/work/woods/m-NetBSD-5/sys/kern/kern_synch.c: No such file or 
directory.
        in /rest/work/woods/m-NetBSD-5/sys/kern/kern_synch.c
(gdb) where
#0  0xc04de9c9 in mi_switch (l=0xc0c3bd80) at 
/rest/work/woods/m-NetBSD-5/sys/kern/kern_synch.c:767
#1  0xc04dac1b in sleepq_block (timo=0, catch=false) at 
/rest/work/woods/m-NetBSD-5/sys/kern/kern_sleepq.c:269
#2  0xc04b4f4c in cv_wait (cv=0xc0d50544, mtx=0xc0d508d8) at 
/rest/work/woods/m-NetBSD-5/sys/kern/kern_condvar.c:201
#3  0xc0462877 in uvm_scheduler () at 
/rest/work/woods/m-NetBSD-5/sys/uvm/uvm_glue.c:550
#4  0xc04a972a in main () at 
/rest/work/woods/m-NetBSD-5/sys/kern/init_main.c:682
(gdb) 

        My version of kern_synch.c has a couple of wee changes related
        to my SYSCALL_STATS experiments so that line number is here:

                /* Switch to the new LWP.. */
767:            prevlwp = cpu_switchto(l, newl, returning);
                ci = curcpu();

        So far as I can tell my changes would not have affected anything
        which should cause this panic.  Indeed this same code ran well
        (though under far less demanding load) on a 2-CPU VirtualBox
        machine for several weeks.

        I can make the kernel image and it's core available, or do any
        other debugging as desired.

        Note I only see that "WARNING: SPL NOT LOWERED ON SYSCALL EXIT"
        message in the panic traceback and in the dump run -- never any
        other time.

>How-To-Repeat:

        run "build.sh -j 4" on an 8-CPU machine

>Fix: