Re: kern/50375: layerfs (nullfs) locking problem leading to livelock

[Jeff Rizzo <riz%NetBSD.org@localhost>]

The following reply was made to PR kern/50375; it has been noted by GNATS.

From: Jeff Rizzo <riz%NetBSD.org@localhost>
To: gnats-bugs%NetBSD.org@localhost, hannken%NetBSD.org@localhost
Cc: 
Subject: Re: kern/50375: layerfs (nullfs) locking problem leading to livelock
Date: Mon, 9 Nov 2015 12:13:47 -0800

 The primary box I'm working stayed alive for just over a week with this 
 patch before crashing in an unrelated way.  However, a different arm box 
 running a kernel with the patch from this PR (also pbulk building), just 
 went into livelock.  Unfortunately, netbsd.gdb appears to be the wrong 
 kernel, so I'm not able to get anything from gdb in this case.  :(
 
 Here's what I did gather (bt/a from all LWPs, 'show lock' on the 
 wchans).  It's not clear to me whether this is also in layerfs, though 
 it would make sense.
 
 armbulk1# ps -axl -oladdr |grep tstile
    12  4849  4947     0 117  0  9808  2696 tstile  D    ?        0:00.45 qmgr -l 926b63c0
 1001 17766 17302 35156 109  0 31544 18688 tstile  D    ?        0:11.60 /usr/li 926b6100
 1001 19955 14589 35156 109  0 23192 10640 tstile  D    ?        0:03.51 /usr/li 91bb9980
 1001 25222 23625 35156 109  0 27416 15400 tstile  D    ?        0:10.25 /usr/li 91b72ba0
 1001 26028 10655 35156 109  0 26392 14304 tstile  D    ?        0:07.68 /usr/li 94f54c60
 
 
 Stopped in pid 0.23 (system) at netbsd:cpu_Debugger+0x4:        bx      r14
 db{2}> bt/a 926b63c0
 trace: pid 4849 lid 1 at 0x9dcabb9c
 0x9dcabb9c: netbsd:mi_switch+0x10
 0x9dcabbcc: netbsd:sleepq_block+0xb4
 0x9dcabc0c: netbsd:turnstile_block+0x318
 0x9dcabc84: netbsd:rw_enter+0x3c0
 0x9dcabcb4: netbsd:genfs_lock+0x68
 0x9dcabcdc: netbsd:VOP_LOCK+0x40
 0x9dcabd04: netbsd:vn_lock+0x88
 0x9dcabe64: netbsd:getcwd_common+0x364
 0x9dcabeb4: netbsd:dostatvfs+0xcc
 0x9dcabee4: netbsd:do_sys_fstatvfs+0x58
 0x9dcabf04: netbsd:sys_fstatvfs1+0x38
 0x9dcabf7c: netbsd:syscall+0xb8
 0x9dcabfac: netbsd:swi_handler+0xa0
 db{2}> bt/a 926b6100
 trace: pid 17766 lid 1 at 0x9c5f3914
 0x9c5f3914: netbsd:mi_switch+0x10
 0x9c5f3944: netbsd:sleepq_block+0xb4
 0x9c5f3984: netbsd:turnstile_block+0x318
 0x9c5f39fc: netbsd:rw_enter+0x3c0
 0x9c5f3a2c: netbsd:genfs_lock+0x68
 0x9c5f3a54: netbsd:VOP_LOCK+0x40
 0x9c5f3a7c: netbsd:vn_lock+0x88
 0x9c5f3bdc: netbsd:getcwd_common+0x364
 0x9c5f3bfc: netbsd:vn_isunder+0x2c
 0x9c5f3c4c: netbsd:lookup_once+0xfc
 0x9c5f3d1c: netbsd:namei_tryemulroot+0x528
 0x9c5f3d54: netbsd:namei+0x34
 0x9c5f3e2c: netbsd:vn_open+0x94
 0x9c5f3eac: netbsd:do_open+0xb0
 0x9c5f3edc: netbsd:do_sys_openat+0x7c
 0x9c5f3f04: netbsd:sys_open+0x38
 0x9c5f3f7c: netbsd:syscall+0xb8
 0x9c5f3fac: netbsd:swi_handler+0xa0
 db{2}> bt/a 91bb9980
 trace: pid 19955 lid 1 at 0x9c5f5914
 0x9c5f5914: netbsd:mi_switch+0x10
 0x9c5f5944: netbsd:sleepq_block+0xb4
 0x9c5f5984: netbsd:turnstile_block+0x318
 0x9c5f59fc: netbsd:rw_enter+0x3c0
 0x9c5f5a2c: netbsd:genfs_lock+0x68
 0x9c5f5a54: netbsd:VOP_LOCK+0x40
 0x9c5f5a7c: netbsd:vn_lock+0x88
 0x9c5f5bdc: netbsd:getcwd_common+0x364
 0x9c5f5bfc: netbsd:vn_isunder+0x2c
 0x9c5f5c4c: netbsd:lookup_once+0xfc
 0x9c5f5d1c: netbsd:namei_tryemulroot+0x528
 0x9c5f5d54: netbsd:namei+0x34
 0x9c5f5e2c: netbsd:vn_open+0x94
 0x9c5f5eac: netbsd:do_open+0xb0
 0x9c5f5edc: netbsd:do_sys_openat+0x7c
 0x9c5f5f04: netbsd:sys_open+0x38
 0x9c5f5f7c: netbsd:syscall+0xb8
 0x9c5f5fac: netbsd:swi_handler+0xa0
 db{2}> bt/a 91b72ba0
 trace: pid 25222 lid 1 at 0x9c6ab914
 0x9c6ab914: netbsd:mi_switch+0x10
 0x9c6ab944: netbsd:sleepq_block+0xb4
 0x9c6ab984: netbsd:turnstile_block+0x318
 0x9c6ab9fc: netbsd:rw_enter+0x3c0
 0x9c6aba2c: netbsd:genfs_lock+0x68
 0x9c6aba54: netbsd:VOP_LOCK+0x40
 0x9c6aba7c: netbsd:vn_lock+0x88
 0x9c6abbdc: netbsd:getcwd_common+0x364
 0x9c6abbfc: netbsd:vn_isunder+0x2c
 0x9c6abc4c: netbsd:lookup_once+0xfc
 0x9c6abd1c: netbsd:namei_tryemulroot+0x528
 0x9c6abd54: netbsd:namei+0x34
 0x9c6abe2c: netbsd:vn_open+0x94
 0x9c6abeac: netbsd:do_open+0xb0
 0x9c6abedc: netbsd:do_sys_openat+0x7c
 0x9c6abf04: netbsd:sys_open+0x38
 0x9c6abf7c: netbsd:syscall+0xb8
 0x9c6abfac: netbsd:swi_handler+0xa0
 db{2}> bt/a 94f54c60
 trace: pid 26028 lid 1 at 0x9c2e1914
 0x9c2e1914: netbsd:mi_switch+0x10
 0x9c2e1944: netbsd:sleepq_block+0xb4
 0x9c2e1984: netbsd:turnstile_block+0x318
 0x9c2e19fc: netbsd:rw_enter+0x3c0
 0x9c2e1a2c: netbsd:genfs_lock+0x68
 0x9c2e1a54: netbsd:VOP_LOCK+0x40
 0x9c2e1a7c: netbsd:vn_lock+0x88
 0x9c2e1bdc: netbsd:getcwd_common+0x364
 0x9c2e1bfc: netbsd:vn_isunder+0x2c
 0x9c2e1c4c: netbsd:lookup_once+0xfc
 0x9c2e1d1c: netbsd:namei_tryemulroot+0x528
 0x9c2e1d54: netbsd:namei+0x34
 0x9c2e1e2c: netbsd:vn_open+0x94
 0x9c2e1eac: netbsd:do_open+0xb0
 0x9c2e1edc: netbsd:do_sys_openat+0x7c
 0x9c2e1f04: netbsd:sys_open+0x38
 0x9c2e1f7c: netbsd:syscall+0xb8
 0x9c2e1fac: netbsd:swi_handler+0xa0
 
 
 ps/w:
 19955    1          cc1plus   netbsd   26 tstile       92ffc4d4
 26028    1          cc1plus   netbsd   26 tstile       92ffc4d4
 25222    1          cc1plus   netbsd   26 tstile       92ffc4d4
 17766    1          cc1plus   netbsd   26 tstile       92ffc4d4
 4849     1             qmgr   netbsd   43 tstile       926cbb04
 
 db{2}> show lock 92ffc4d4
 lock address : 0x0000000092ffc4d4 type     :     sleep/adaptive
 initialized  : 0x0000000081365434
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  4
 current cpu  :                  2 last held:                  1
 current lwp  : 0x00000000915aa020 last held: 0x0000000096ddc9e0
 last locked* : 0x000000008137a71c unlocked : 0x000000008137a838
 owner/count  : 0x0000000096ddc9e0 flags    : 0x0000000000000007
 
 Turnstile chain at 0x815eaed0.
 => Turnstile at 0x951e5230 (wrq=0x951e5240, rdq=0x951e5248).
 => 0 waiting readers:
 => 4 waiting writers: 0x91bb9980 0x94f54c60 0x91b72ba0 0x926b6100
 db{2}> show lock 926cbb04
 lock address : 0x00000000926cbb04 type     :     sleep/adaptive
 initialized  : 0x0000000081365434
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  1
 current cpu  :                  2 last held:                  1
 current lwp  : 0x00000000915aa020 last held: 0x0000000096ddd4e0
 last locked* : 0x000000008137a71c unlocked : 0x000000008137a838
 owner/count  : 0x0000000096ddd4e0 flags    : 0x0000000000000007
 
 Turnstile chain at 0x815eaf00.
 => Turnstile at 0x9159acb0 (wrq=0x9159acc0, rdq=0x9159acc8).
 => 0 waiting readers:
 => 1 waiting writers: 0x926b63c0
 db{2}> bt/a 0x0000000096ddc9e0
 trace: pid 4848 lid 1 at 0x9c53b5c4
 0x9c53b5c4: netbsd:mi_switch+0x10
 0x9c53b5f4: netbsd:sleepq_block+0xb4
 0x9c53b62c: netbsd:cv_wait+0x130
 0x9c53b6b4: netbsd:vmem_xalloc+0x504
 0x9c53b6f4: netbsd:vmem_alloc+0xe8
 0x9c53b77c: netbsd:vmem_xalloc+0x6a8
 0x9c53b7bc: netbsd:vmem_alloc+0xe8
 0x9c53b7ec: netbsd:qc_poolpage_alloc+0x54
 0x9c53b82c: netbsd:pool_grow+0x38
 0x9c53b864: netbsd:pool_get+0x80
 0x9c53b8ac: netbsd:pool_cache_get_slow+0x224
 0x9c53b8e4: netbsd:pool_cache_get_paddr+0x22c
 0x9c53b924: netbsd:vmem_alloc+0x90
 0x9c53b974: netbsd:uvm_km_kmem_alloc+0x38
 0x9c53b98c: netbsd:pool_page_alloc+0x3c
 0x9c53b9cc: netbsd:pool_grow+0x38
 0x9c53ba04: netbsd:pool_get+0x80
 0x9c53ba4c: netbsd:pool_cache_get_slow+0x224
 0x9c53ba84: netbsd:pool_cache_get_paddr+0x22c
 0x9c53baa4: netbsd:vnalloc+0x2c
 0x9c53bb0c: netbsd:vcache_get+0x224
 0x9c53bbc4: netbsd:ufs_lookup+0x858
 0x9c53bbfc: netbsd:VOP_LOOKUP+0x48
 0x9c53bc4c: netbsd:lookup_once+0x19c
 0x9c53bd1c: netbsd:namei_tryemulroot+0x528
 0x9c53bd54: netbsd:namei+0x34
 0x9c53be2c: netbsd:vn_open+0x94
 0x9c53beac: netbsd:do_open+0xb0
 0x9c53bedc: netbsd:do_sys_openat+0x7c
 0x9c53bf04: netbsd:sys_open+0x38
 0x9c53bf7c: netbsd:syscall+0xb8
 0x9c53bfac: netbsd:swi_handler+0xa0
 db{2}> bt/a 0x0000000096ddd4e0
 trace: pid 7394 lid 1 at 0x9c471724
 0x9c471724: netbsd:mi_switch+0x10
 0x9c471754: netbsd:sleepq_block+0xb4
 0x9c47178c: netbsd:cv_wait+0x130
 0x9c471814: netbsd:vmem_xalloc+0x504
 0x9c471854: netbsd:vmem_alloc+0xe8
 0x9c4718dc: netbsd:vmem_xalloc+0x6a8
 0x9c47191c: netbsd:vmem_alloc+0xe8
 0x9c47194c: netbsd:qc_poolpage_alloc+0x54
 0x9c47198c: netbsd:pool_grow+0x38
 0x9c4719c4: netbsd:pool_get+0x80
 0x9c471a0c: netbsd:pool_cache_get_slow+0x224
 0x9c471a44: netbsd:pool_cache_get_paddr+0x22c
 0x9c471a84: netbsd:vmem_alloc+0x90
 0x9c471ad4: netbsd:uvm_km_kmem_alloc+0x38
 0x9c471aec: netbsd:pool_page_alloc+0x3c
 0x9c471b2c: netbsd:pool_grow+0x38
 0x9c471b64: netbsd:pool_get+0x80
 0x9c471bac: netbsd:pool_cache_get_slow+0x224
 0x9c471be4: netbsd:pool_cache_get_paddr+0x22c
 0x9c471c2c: netbsd:kmem_intr_alloc+0x7c
 0x9c471ccc: netbsd:ufs_readdir+0x12c
 0x9c471d04: netbsd:VOP_READDIR+0x58
 0x9c471e64: netbsd:getcwd_common+0x428
 0x9c471eb4: netbsd:dostatvfs+0xcc
 0x9c471ee4: netbsd:do_sys_fstatvfs+0x58
 0x9c471f04: netbsd:sys_fstatvfs1+0x38
 0x9c471f7c: netbsd:syscall+0xb8
 0x9c471fac: netbsd:swi_handler+0xa0
 db{2}>