Re: Help with ddb and lockdebug.

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

Brian Buhrow wrote:
>         Hello.  I'm trying to debug akernel module I'm working on which is
> having locking issues.  I have a 5.0 kernel with  options LOCKDEBUG turned
> on, and I'm able to figure out some of what I'm seeing, but I have a couple
> of questions, and I hope someone here can help point me in the right
> direction so I can make further progress on getting zaptel working on
> NetBSD-5.x.  
> I'm seeing output like the following and I have a couple of questions:
>
> Mutex error: lockdebug_barrier: spin lock held

The CPU is probably trying to hold more than one spinlock, which can 
result in nasty deadlocks. Hence, the panic.

> lock address : 0x00000000c0b24700 type     :               spin
> initialized  : 0x00000000c043a946
> shared holds :                  0 exclusive:                  1
> shares wanted:                  0 exclusive:                  0
> current cpu  :                  1 last held:                  1
> current lwp  : 0x00000000ce0a7520 last held: 0x00000000ce0a7520
> last locked  : 0x00000000c043a8a6 unlocked : 0x00000000c04280f3
> owner field  : 0x0000000000010700 wait/spin:                0/1
>
>         1.   What are the pointers for the locked and unlocked fields?  Are
> they pointers to the struct proc structures for the last processes that
> locked and unlocked this lock respectively?  If so, how can I examine those
> proc structures in ddb?

(provided without warranty)

- lock address: where the lock (spin mutex) resides in virtual memory
- initialized: where (in code) it was initialized
- last locked: where it was acquired for the last time
- unlocked: similar to 'last locked', this time for unlocking
- current lwp: the pointer to the lwp struct that acquired the mutex 
when the lockdebug fired

To obtain information about the lwp in ddb:
$ ps/l # will list the different threads, look for the offending one
$ bt/a <lwp_addr> # for backtrace of the associated lwp

See ddb(4)

> 2.  How do I read the owner field?  It doesn't look like a uid or a pid,
> what is it?

For adaptive mutexes, it stores a pointer to the current owner of the 
mutex. For spin mutexes, there is no "ownership" context (spin mutexes 
are hold for a very short amount of time, and do not sleep), it is just 
used as a bit field.

See kern/kern_mutex.c and mutex(9).

> 3.  Is there documentation on some of this somewhere?  I've read the ddb
> man page, and I'm familiar with the 4.4BSD internals book by Kirk Mcusick.

NetBSD is moving to a fine-grained locking model, which is quite 
different from the 4.4BSD one. "Solaris internals" might be closer to 
what you want.

> I've been working with NetBSD for years, but I'm having trouble finding
> where this output comes from in the source tree.

See kern/subr_lockdebug.c

>         In short, if anyone has pointers on how I can get more complete
> information on what I'm looking at in ddb, I would be very interested.

HTH

PS: anyone is welcomed to make additions or corrections, of course.

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost