zvol related crash

[Staffan Thomén <duck%shangtai.net@localhost>]

Due to our newfound ZFS powers, I wanted to play around with it and see if it 
would work as a replacement for lvm on the xen dom0.

This weekend I upgraded my dom0 to 9.0 and it was a breeze. I also took care 
to load the zfs module and that also worked without issue.

Yesterday I got some more disks and set up a raidz zpool, initialized a volume 
and exported it using iscsi, holding a ntfs filesystem.

Everything worked wonderfully.

Today I created another small zvol and copied one of my existing test domU:s 
disk to it from lvm.

It booted alright in single user with a netbsd-9 kernel, the disk mounted r/w 
ok and I ftp:d the sets from my build server but when I tried to extract the 
sets it pretty quickly paniced and when I tried to reboot the domU, it hosed 
the entire system.

From what I could see being spat out to the console, xen seems to have been 
rebooting all the domU:s, but none of them stayed arond long; I could use the 
IPMI SOL to get to a root shell on the dom0, and running xl list showed me the 
domains running but the next ls -l just hung. At which point I paniced 
slightly and reset the system.

Below's a log excerpt, starting at the point of starting up the test domU with 
one zvol disk and ending at system reset. The zvol is actually called 
internal1/test-root, but the entire name isn't printed for some reason.

The "Unknown operation 21" message stands out to me. I also recall seeing 
"Unknown operation 30" on the domU:s log before it crashed. I didn't have the 
presence of mind to write down the panic message or stack trace, but it was 
something about some lock, perhaps biglock.

There's also xbdback_map_shm: xen_shm error 1 xbd IO domain 27: error 1,
domain 27 is the test domain as it seems to be rebooted.

Did I do something I wasn't supposed to?

Staffan

Feb 25 20:27:51 bluegleam /netbsd: [ 258692.3376185] xbd backend: attach 
device internal1/test- (size 20971520) for domain 23
Feb 25 20:27:51 bluegleam /netbsd: [ 258692.5676145] xvif23i0: Ethernet 
address 00:16:3e:01:07:06
Feb 25 20:27:53 bluegleam /netbsd: [ 258694.5775921] xbd backend domain 23 
handle 0x300 (768) using event channel 48, protocol x86_64-abi
Feb 25 20:27:55 bluegleam ntpd[565]: Listen normally on 23 xvif23i0 
[fe80::216:3eff:fe01:706%16]:123
Feb 25 20:30:10 bluegleam /netbsd: [ 258831.4759716] in6_ifadd: 
2001:470:de9f:1:ec4:7aff:fe03:610 is already configured
Feb 25 20:30:13 bluegleam ntpd[565]: Listen normally on 24 xvif23i0 
[2001:470:de9f:1:216:3eff:fe01:706]:123
Feb 25 20:31:56 bluegleam /netbsd: [ 258925.7248116] xbdb23i768: unknown 
operation 21
Feb 25 20:31:56 bluegleam /netbsd: [ 258937.4846724] xbd backend: detach 
device vm-power for domain 5
Feb 25 20:31:56 bluegleam /netbsd: [ 258937.5746697] xvif5i0: disconnecting
Feb 25 20:31:56 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:31:59 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:31:59 bluegleam ntpd[565]: Deleting interface #18 xvif5i0, 
fe80::216:3eff:fe01:703%13#123, interface stats: received=0, sent=0, 
dropped=0, active_time=258900 secs
Feb 25 20:31:59 bluegleam ntpd[565]: Deleting interface #19 xvif5i0, 
2001:470:de9f:1:216:3eff:fe01:703#123, interface stats: received=0, sent=0, 
dropped=0, active_time=258894 secs
Feb 25 20:32:01 bluegleam /netbsd: [ 258941.7546191] xbd backend: detach 
device vm-data for domain 1
Feb 25 20:32:01 bluegleam /netbsd: [ 258941.7946214] xbd backend: detach 
device data-userdata for domain 1
Feb 25 20:32:01 bluegleam /netbsd: [ 258941.8846173] xvif1i0: disconnecting
Feb 25 20:32:01 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:32:02 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:32:02 bluegleam ntpd[565]: Deleting interface #9 xvif1i0, 
fe80::216:3eff:fe01:701%8#123, interface stats: received=0, sent=0, dropped=0, 
active_time=258903 secs
Feb 25 20:32:02 bluegleam ntpd[565]: Deleting interface #10 xvif1i0, 
2001:470:de9f:1:216:3eff:fe01:701#123, interface stats: received=0, sent=0, 
dropped=0, active_time=258903 secs
Feb 25 20:32:03 bluegleam /netbsd: [ 258943.5845981] xbd backend: attach 
device vm-power (size 209715200) for domain 24
Feb 25 20:32:03 bluegleam /netbsd: [ 258943.8045960] xvif24i0: Ethernet 
address 00:16:3e:01:07:03
Feb 25 20:32:04 bluegleam /netbsd: [ 258944.7845826] xbd backend: attach 
device vm-data (size 20971520) for domain 25
Feb 25 20:32:04 bluegleam /netbsd: [ 258944.8345826] xbd backend: attach 
device data-userdata (size 1048576000) for domain 25
Feb 25 20:32:04 bluegleam /netbsd: [ 258945.0345831] xvif25i0: Ethernet 
address 00:16:3e:01:07:01
Feb 25 20:32:04 bluegleam /netbsd: [ 258945.7145722] xbd backend domain 25 
handle 0x300 (768) using event channel 50, protocol x86_64-abi
Feb 25 20:32:04 bluegleam /netbsd: [ 258945.7245712] xbd backend domain 25 
handle 0x340 (832) using event channel 51, protocol x86_64-abi
Feb 25 20:32:05 bluegleam ntpd[565]: Listen normally on 25 xvif25i0 
[fe80::216:3eff:fe01:701%18]:123
Feb 25 20:32:06 bluegleam /netbsd: [ 258947.7145479] xbd backend domain 24 
handle 0x300 (768) using event channel 53, protocol x86_64-abi
Feb 25 20:32:15 bluegleam /netbsd: [ 258955.7844507] xbd backend: detach 
device vm-mail for domain 2
Feb 25 20:32:15 bluegleam /netbsd: [ 258955.8944501] xvif2i0: disconnecting
Feb 25 20:32:15 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:32:16 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:32:16 bluegleam ntpd[565]: Listen normally on 26 xvif24i0 
[fe80::216:3eff:fe01:703%17]:123
Feb 25 20:32:16 bluegleam ntpd[565]: Deleting interface #11 xvif2i0, 
fe80::216:3eff:fe01:702%9#123, interface stats: received=0, sent=0, dropped=0, 
active_time=258917 secs
Feb 25 20:32:16 bluegleam ntpd[565]: Deleting interface #12 xvif2i0, 
2001:470:de9f:1:216:3eff:fe01:702#123, interface stats: received=0, sent=0, 
dropped=0, active_time=258917 secs
Feb 25 20:32:18 bluegleam /netbsd: [ 258958.7144172] xbd backend: attach 
device vm-mail (size 20971520) for domain 26
Feb 25 20:32:18 bluegleam /netbsd: [ 258958.9744128] xvif26i0: Ethernet 
address 00:16:3e:01:07:02
Feb 25 20:32:18 bluegleam /netbsd: [ 258959.7244050] xbd backend domain 26 
handle 0x300 (768) using event channel 55, protocol x86_64-abi
Feb 25 20:32:19 bluegleam ntpd[565]: Listen normally on 27 xvif26i0 
[fe80::216:3eff:fe01:702%19]:123
Feb 25 20:35:06 bluegleam /netbsd: [ 259126.7423969] xbd backend: detach 
device internal1/test- for domain 23
Feb 25 20:35:06 bluegleam /netbsd: [ 259126.8623955] xvif23i0: disconnecting
Feb 25 20:35:06 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:35:07 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:35:07 bluegleam ntpd[565]: Deleting interface #23 xvif23i0, 
fe80::216:3eff:fe01:706%16#123, interface stats: received=0, sent=0, 
dropped=0, active_time=431 secs
Feb 25 20:35:07 bluegleam ntpd[565]: Deleting interface #24 xvif23i0, 
2001:470:de9f:1:216:3eff:fe01:706#123, interface stats: received=0, sent=0, 
dropped=0, active_time=293 secs
Feb 25 20:35:09 bluegleam /netbsd: [ 259129.5823619] xbd backend: attach 
device internal1/test- (size 20971520) for domain 27
Feb 25 20:35:09 bluegleam /netbsd: [ 259129.8323594] xvif27i0: Ethernet 
address 00:16:3e:01:07:06
Feb 25 20:35:09 bluegleam /netbsd: [ 259130.7123489] xbd backend domain 27 
handle 0x300 (768) using event channel 57, protocol x86_64-abi
Feb 25 20:35:10 bluegleam ntpd[565]: Listen normally on 28 xvif27i0 
[fe80::216:3eff:fe01:706%20]:123
Feb 25 20:35:55 bluegleam /netbsd: [ 259166.8619141] xbdback_map_shm: xen_shm 
error 1 xbd IO domain 27: error 1
Feb 25 20:35:55 bluegleam /netbsd: [ 259175.7818082] xbd backend: detach 
device vm-shell for domain 3
Feb 25 20:35:55 bluegleam /netbsd: [ 259175.8818108] xvif3i0: disconnecting
Feb 25 20:35:55 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:35:55 bluegleam /netbsd: [ 259175.9418063] xvif3i1: disconnecting
Feb 25 20:35:56 bluegleam syslogd[285]: last message repeated 5 times
Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #13 xvif3i0, 
fe80::216:3eff:fe01:704%10#123, interface stats: received=0, sent=0, 
dropped=0, active_time=259137 secs
Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #14 xvif3i0, 
2001:470:de9f:1:216:3eff:fe01:704#123, interface stats: received=1024, 
sent=1030, dropped=0, active_time=259137 secs
Feb 25 20:35:56 bluegleam ntpd[565]: 2a01:4f9:2a:1919::9302 local addr 
2001:470:de9f:1:216:3eff:fe01:704 -> <null>
Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #15 xvif3i1, 
fe80::216:3eff:fe01:805%11#123, interface stats: received=0, sent=0, 
dropped=0, active_time=259137 secs
Feb 25 20:35:58 bluegleam /netbsd: [ 259178.7017732] xbd backend: attach 
device vm-shell (size 10485760) for domain 28
Feb 25 20:35:58 bluegleam /netbsd: [ 259178.9517708] xvif28i0: Ethernet 
address 00:16:3e:01:07:04
Feb 25 20:35:58 bluegleam /netbsd: [ 259179.0917688] xvif28i1: Ethernet 
address 00:16:3e:01:08:05
Feb 25 20:35:58 bluegleam /netbsd: [ 259179.7217603] xbd backend domain 28 
handle 0x300 (768) using event channel 59, protocol x86_64-abi
Feb 25 20:35:59 bluegleam ntpd[565]: Listen normally on 29 xvif28i0 
[fe80::216:3eff:fe01:704%21]:123
Feb 25 20:35:59 bluegleam ntpd[565]: Listen normally on 30 xvif28i1 
[fe80::216:3eff:fe01:805%22]:123
Feb 25 20:37:09 bluegleam /netbsd: [ 259249.7609203] xbd backend: detach 
device vm-www for domain 4
Feb 25 20:37:09 bluegleam /netbsd: [ 259249.8609194] xvif4i0: disconnecting
Feb 25 20:37:09 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:37:10 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:37:10 bluegleam ntpd[565]: Deleting interface #16 xvif4i0, 
fe80::216:3eff:fe01:705%12#123, interface stats: received=0, sent=0, 
dropped=0, active_time=259211 secs
Feb 25 20:37:10 bluegleam ntpd[565]: Deleting interface #17 xvif4i0, 
2001:470:de9f:1:216:3eff:fe01:705#123, interface stats: received=0, sent=0, 
dropped=0, active_time=259211 secs
Feb 25 20:37:12 bluegleam /netbsd: [ 259252.6908885] xbd backend: attach 
device vm-www (size 20971520) for domain 29
Feb 25 20:37:12 bluegleam /netbsd: [ 259252.9408814] xvif29i0: Ethernet 
address 00:16:3e:01:07:05
Feb 25 20:37:12 bluegleam /netbsd: [ 259253.7508729] xbd backend domain 29 
handle 0x300 (768) using event channel 62, protocol x86_64-abi
Feb 25 20:37:13 bluegleam ntpd[565]: Listen normally on 31 xvif29i0 
[fe80::216:3eff:fe01:705%23]:123
Feb 25 20:37:20 bluegleam /netbsd: [ 259261.4907822] in6_ifadd: 
2001:470:de9f:1:ec4:7aff:fe03:610 is already configured
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 32 xvif24i0 
[2001:470:de9f:1:216:3eff:fe01:703]:123
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 33 xvif25i0 
[2001:470:de9f:1:216:3eff:fe01:701]:123
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 34 xvif26i0 
[2001:470:de9f:1:216:3eff:fe01:702]:123
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 35 xvif27i0 
[2001:470:de9f:1:216:3eff:fe01:706]:123
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 36 xvif28i0 
[2001:470:de9f:1:216:3eff:fe01:704]:123
Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 37 xvif29i0 
[2001:470:de9f:1:216:3eff:fe01:705]:123
Feb 25 20:37:56 bluegleam /netbsd: [ 259297.6503468] xbd backend: detach 
device internal1/test- for domain 27
Feb 25 20:37:56 bluegleam /netbsd: [ 259297.7503457] xvif27i0: disconnecting
Feb 25 20:37:56 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
configured
Feb 25 20:38:16 bluegleam syslogd[285]: last message repeated 3 times
Feb 25 20:38:16 bluegleam ntpd[565]: Deleting interface #28 xvif27i0, 
fe80::216:3eff:fe01:706%20#123, interface stats: received=0, sent=0, 
dropped=0, active_time=186 secs
Feb 25 20:38:16 bluegleam ntpd[565]: Deleting interface #35 xvif27i0, 
2001:470:de9f:1:216:3eff:fe01:706#123, interface stats: received=0, sent=0, 
dropped=0, active_time=53 secs
Feb 25 20:38:49 bluegleam /netbsd: [ 259350.6097125] nfs server data:/data: 
not responding