NetBSD DomU MP freeze under Linux Dom0

[Roger Pau Monne <roger.pau%citrix.com@localhost>]

Hello,

Recently I've been doing some benchmarks on NetBSD, to compare the
performances of both NetBSD and Linux as Dom0/DomUs (this was presented
on XenSummit last week with Cherry G. Mathew, slides will probably be
uploaded soon).

One of the benchmarks consisted in running build.sh inside a DomU, and
during this test I've realised that this lead to a freeze when running a
Linux Dom0 and a NetBSD DomU with 4vcpus. So far I haven't been able to
reproduce the problem without MP or in a NetBSD Dom0, which is kind of
strange, because I would say it is not related to blkfront, I've added
some debugging prints there, and blkfront seems to not be the owner of
the lock when the freeze happens. The build of NetBSD inside the DomU
was using 8 simultaneous jobs, and it freezes to a point where I can not
even access ddb. I've been able to get a trace using gdbsx:

Thread 4:

#0  0xffffffff80101248 in hypercall_page ()
#1  0x000000000000e033 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3:

#0  0xffffffff80130f32 in x86_pause ()
#1  0xffffffff801f67b1 in _kernel_lock ()
#2  0xffffffff8030b054 in bdev_strategy ()
#3  0xffffffff803037d8 in spec_strategy ()
#4  0xffffffff803a719a in VOP_STRATEGY ()
#5  0xffffffff8035ff7a in ufs_strategy ()
#6  0xffffffff803a719a in VOP_STRATEGY ()
#7  0xffffffff8038d3fa in bwrite ()
#8  0xffffffff803a6320 in VOP_BWRITE ()
#9  0xffffffff80357125 in ufs_dirremove ()
#10 0xffffffff8035dc47 in ufs_remove ()
#11 0xffffffff803a6b53 in VOP_REMOVE ()
#12 0xffffffff8039ac4f in do_sys_unlink ()
#13 0xffffffff8032b044 in syscall ()
#14 0xffffffff8010221d in Xsyscall ()

Thread 2:

#0  0xffffffff801f67b1 in _kernel_lock ()
#1  0xffffffff8030b054 in bdev_strategy ()
#2  0xffffffff803037d8 in spec_strategy ()
#3  0xffffffff803a719a in VOP_STRATEGY ()
#4  0xffffffff8035ff7a in ufs_strategy ()
#5  0xffffffff803a719a in VOP_STRATEGY ()
#6  0xffffffff8038d3fa in bwrite ()
#7  0xffffffff803a6320 in VOP_BWRITE ()
#8  0xffffffff80357125 in ufs_dirremove ()
#9  0xffffffff8035dc47 in ufs_remove ()
#10 0xffffffff803a6b53 in VOP_REMOVE ()
#11 0xffffffff8039ac4f in do_sys_unlink ()
#12 0xffffffff8032b044 in syscall ()
#13 0xffffffff8010221d in Xsyscall ()

Thread 1:

#0  0xffffffff801f67b1 in _kernel_lock ()
#1  0xffffffff8030b054 in bdev_strategy ()
#2  0xffffffff803037d8 in spec_strategy ()
#3  0xffffffff803a719a in VOP_STRATEGY ()
#4  0xffffffff8035ff7a in ufs_strategy ()
#5  0xffffffff803a719a in VOP_STRATEGY ()
#6  0xffffffff8038d3fa in bwrite ()
#7  0xffffffff803a6320 in VOP_BWRITE ()
#8  0xffffffff80357125 in ufs_dirremove ()
#9  0xffffffff8035dc47 in ufs_remove ()
#10 0xffffffff803a6b53 in VOP_REMOVE ()
#11 0xffffffff8039ac4f in do_sys_unlink ()
#12 0xffffffff8032b044 in syscall ()
#13 0xffffffff8010221d in Xsyscall ()

My guess is that Thread 4 is holding the lock, and it's blocked for some
reason that's beyond my current knowledge of NetBSD internals, and the
stack trace is not helping on that.

Any help in debugging this would be appreciated.

Roger.