Re: zvol related crash

[Brian Buhrow <buhrow%nfbcal.org@localhost>]

	hello.  This, I believe, is the same as bug kern/54724, a bug I filed
in November of 2019.   Please add your findings to that bug.
Zfs is corrupting kernel memory, causing a lot of seemingly unrelated
crashes.

-thanks
-Brian

On Feb 25,  9:32pm, =?UTF-8?B?U3RhZmZhbiBUaG9tw6lu?= wrote:
} Subject: zvol related crash
} Due to our newfound ZFS powers, I wanted to play around with it and see if it 
} would work as a replacement for lvm on the xen dom0.
} 
} This weekend I upgraded my dom0 to 9.0 and it was a breeze. I also took care 
} to load the zfs module and that also worked without issue.
} 
} Yesterday I got some more disks and set up a raidz zpool, initialized a volume 
} and exported it using iscsi, holding a ntfs filesystem.
} 
} Everything worked wonderfully.
} 
} Today I created another small zvol and copied one of my existing test domU:s 
} disk to it from lvm.
} 
} It booted alright in single user with a netbsd-9 kernel, the disk mounted r/w 
} ok and I ftp:d the sets from my build server but when I tried to extract the 
} sets it pretty quickly paniced and when I tried to reboot the domU, it hosed 
} the entire system.
} 
}  From what I could see being spat out to the console, xen seems to have been 
} rebooting all the domU:s, but none of them stayed arond long; I could use the 
} IPMI SOL to get to a root shell on the dom0, and running xl list showed me the 
} domains running but the next ls -l just hung. At which point I paniced 
} slightly and reset the system.
} 
} Below's a log excerpt, starting at the point of starting up the test domU with 
} one zvol disk and ending at system reset. The zvol is actually called 
} internal1/test-root, but the entire name isn't printed for some reason.
} 
} The "Unknown operation 21" message stands out to me. I also recall seeing 
} "Unknown operation 30" on the domU:s log before it crashed. I didn't have the 
} presence of mind to write down the panic message or stack trace, but it was 
} something about some lock, perhaps biglock.
} 
} There's also xbdback_map_shm: xen_shm error 1 xbd IO domain 27: error 1,
} domain 27 is the test domain as it seems to be rebooted.
} 
} Did I do something I wasn't supposed to?
} 
} Staffan
} 
} Feb 25 20:27:51 bluegleam /netbsd: [ 258692.3376185] xbd backend: attach 
} device internal1/test- (size 20971520) for domain 23
} Feb 25 20:27:51 bluegleam /netbsd: [ 258692.5676145] xvif23i0: Ethernet 
} address 00:16:3e:01:07:06
} Feb 25 20:27:53 bluegleam /netbsd: [ 258694.5775921] xbd backend domain 23 
} handle 0x300 (768) using event channel 48, protocol x86_64-abi
} Feb 25 20:27:55 bluegleam ntpd[565]: Listen normally on 23 xvif23i0 
} [fe80::216:3eff:fe01:706%16]:123
} Feb 25 20:30:10 bluegleam /netbsd: [ 258831.4759716] in6_ifadd: 
} 2001:470:de9f:1:ec4:7aff:fe03:610 is already configured
} Feb 25 20:30:13 bluegleam ntpd[565]: Listen normally on 24 xvif23i0 
} [2001:470:de9f:1:216:3eff:fe01:706]:123
} Feb 25 20:31:56 bluegleam /netbsd: [ 258925.7248116] xbdb23i768: unknown 
} operation 21
} Feb 25 20:31:56 bluegleam /netbsd: [ 258937.4846724] xbd backend: detach 
} device vm-power for domain 5
} Feb 25 20:31:56 bluegleam /netbsd: [ 258937.5746697] xvif5i0: disconnecting
} Feb 25 20:31:56 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:31:59 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:31:59 bluegleam ntpd[565]: Deleting interface #18 xvif5i0, 
} fe80::216:3eff:fe01:703%13#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=258900 secs
} Feb 25 20:31:59 bluegleam ntpd[565]: Deleting interface #19 xvif5i0, 
} 2001:470:de9f:1:216:3eff:fe01:703#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=258894 secs
} Feb 25 20:32:01 bluegleam /netbsd: [ 258941.7546191] xbd backend: detach 
} device vm-data for domain 1
} Feb 25 20:32:01 bluegleam /netbsd: [ 258941.7946214] xbd backend: detach 
} device data-userdata for domain 1
} Feb 25 20:32:01 bluegleam /netbsd: [ 258941.8846173] xvif1i0: disconnecting
} Feb 25 20:32:01 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:32:02 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:32:02 bluegleam ntpd[565]: Deleting interface #9 xvif1i0, 
} fe80::216:3eff:fe01:701%8#123, interface stats: received=0, sent=0, dropped=0, 
} active_time=258903 secs
} Feb 25 20:32:02 bluegleam ntpd[565]: Deleting interface #10 xvif1i0, 
} 2001:470:de9f:1:216:3eff:fe01:701#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=258903 secs
} Feb 25 20:32:03 bluegleam /netbsd: [ 258943.5845981] xbd backend: attach 
} device vm-power (size 209715200) for domain 24
} Feb 25 20:32:03 bluegleam /netbsd: [ 258943.8045960] xvif24i0: Ethernet 
} address 00:16:3e:01:07:03
} Feb 25 20:32:04 bluegleam /netbsd: [ 258944.7845826] xbd backend: attach 
} device vm-data (size 20971520) for domain 25
} Feb 25 20:32:04 bluegleam /netbsd: [ 258944.8345826] xbd backend: attach 
} device data-userdata (size 1048576000) for domain 25
} Feb 25 20:32:04 bluegleam /netbsd: [ 258945.0345831] xvif25i0: Ethernet 
} address 00:16:3e:01:07:01
} Feb 25 20:32:04 bluegleam /netbsd: [ 258945.7145722] xbd backend domain 25 
} handle 0x300 (768) using event channel 50, protocol x86_64-abi
} Feb 25 20:32:04 bluegleam /netbsd: [ 258945.7245712] xbd backend domain 25 
} handle 0x340 (832) using event channel 51, protocol x86_64-abi
} Feb 25 20:32:05 bluegleam ntpd[565]: Listen normally on 25 xvif25i0 
} [fe80::216:3eff:fe01:701%18]:123
} Feb 25 20:32:06 bluegleam /netbsd: [ 258947.7145479] xbd backend domain 24 
} handle 0x300 (768) using event channel 53, protocol x86_64-abi
} Feb 25 20:32:15 bluegleam /netbsd: [ 258955.7844507] xbd backend: detach 
} device vm-mail for domain 2
} Feb 25 20:32:15 bluegleam /netbsd: [ 258955.8944501] xvif2i0: disconnecting
} Feb 25 20:32:15 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:32:16 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:32:16 bluegleam ntpd[565]: Listen normally on 26 xvif24i0 
} [fe80::216:3eff:fe01:703%17]:123
} Feb 25 20:32:16 bluegleam ntpd[565]: Deleting interface #11 xvif2i0, 
} fe80::216:3eff:fe01:702%9#123, interface stats: received=0, sent=0, dropped=0, 
} active_time=258917 secs
} Feb 25 20:32:16 bluegleam ntpd[565]: Deleting interface #12 xvif2i0, 
} 2001:470:de9f:1:216:3eff:fe01:702#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=258917 secs
} Feb 25 20:32:18 bluegleam /netbsd: [ 258958.7144172] xbd backend: attach 
} device vm-mail (size 20971520) for domain 26
} Feb 25 20:32:18 bluegleam /netbsd: [ 258958.9744128] xvif26i0: Ethernet 
} address 00:16:3e:01:07:02
} Feb 25 20:32:18 bluegleam /netbsd: [ 258959.7244050] xbd backend domain 26 
} handle 0x300 (768) using event channel 55, protocol x86_64-abi
} Feb 25 20:32:19 bluegleam ntpd[565]: Listen normally on 27 xvif26i0 
} [fe80::216:3eff:fe01:702%19]:123
} Feb 25 20:35:06 bluegleam /netbsd: [ 259126.7423969] xbd backend: detach 
} device internal1/test- for domain 23
} Feb 25 20:35:06 bluegleam /netbsd: [ 259126.8623955] xvif23i0: disconnecting
} Feb 25 20:35:06 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:35:07 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:35:07 bluegleam ntpd[565]: Deleting interface #23 xvif23i0, 
} fe80::216:3eff:fe01:706%16#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=431 secs
} Feb 25 20:35:07 bluegleam ntpd[565]: Deleting interface #24 xvif23i0, 
} 2001:470:de9f:1:216:3eff:fe01:706#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=293 secs
} Feb 25 20:35:09 bluegleam /netbsd: [ 259129.5823619] xbd backend: attach 
} device internal1/test- (size 20971520) for domain 27
} Feb 25 20:35:09 bluegleam /netbsd: [ 259129.8323594] xvif27i0: Ethernet 
} address 00:16:3e:01:07:06
} Feb 25 20:35:09 bluegleam /netbsd: [ 259130.7123489] xbd backend domain 27 
} handle 0x300 (768) using event channel 57, protocol x86_64-abi
} Feb 25 20:35:10 bluegleam ntpd[565]: Listen normally on 28 xvif27i0 
} [fe80::216:3eff:fe01:706%20]:123
} Feb 25 20:35:55 bluegleam /netbsd: [ 259166.8619141] xbdback_map_shm: xen_shm 
} error 1 xbd IO domain 27: error 1
} Feb 25 20:35:55 bluegleam /netbsd: [ 259175.7818082] xbd backend: detach 
} device vm-shell for domain 3
} Feb 25 20:35:55 bluegleam /netbsd: [ 259175.8818108] xvif3i0: disconnecting
} Feb 25 20:35:55 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:35:55 bluegleam /netbsd: [ 259175.9418063] xvif3i1: disconnecting
} Feb 25 20:35:56 bluegleam syslogd[285]: last message repeated 5 times
} Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #13 xvif3i0, 
} fe80::216:3eff:fe01:704%10#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=259137 secs
} Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #14 xvif3i0, 
} 2001:470:de9f:1:216:3eff:fe01:704#123, interface stats: received=1024, 
} sent=1030, dropped=0, active_time=259137 secs
} Feb 25 20:35:56 bluegleam ntpd[565]: 2a01:4f9:2a:1919::9302 local addr 
} 2001:470:de9f:1:216:3eff:fe01:704 -> <null>
} Feb 25 20:35:56 bluegleam ntpd[565]: Deleting interface #15 xvif3i1, 
} fe80::216:3eff:fe01:805%11#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=259137 secs
} Feb 25 20:35:58 bluegleam /netbsd: [ 259178.7017732] xbd backend: attach 
} device vm-shell (size 10485760) for domain 28
} Feb 25 20:35:58 bluegleam /netbsd: [ 259178.9517708] xvif28i0: Ethernet 
} address 00:16:3e:01:07:04
} Feb 25 20:35:58 bluegleam /netbsd: [ 259179.0917688] xvif28i1: Ethernet 
} address 00:16:3e:01:08:05
} Feb 25 20:35:58 bluegleam /netbsd: [ 259179.7217603] xbd backend domain 28 
} handle 0x300 (768) using event channel 59, protocol x86_64-abi
} Feb 25 20:35:59 bluegleam ntpd[565]: Listen normally on 29 xvif28i0 
} [fe80::216:3eff:fe01:704%21]:123
} Feb 25 20:35:59 bluegleam ntpd[565]: Listen normally on 30 xvif28i1 
} [fe80::216:3eff:fe01:805%22]:123
} Feb 25 20:37:09 bluegleam /netbsd: [ 259249.7609203] xbd backend: detach 
} device vm-www for domain 4
} Feb 25 20:37:09 bluegleam /netbsd: [ 259249.8609194] xvif4i0: disconnecting
} Feb 25 20:37:09 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:37:10 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:37:10 bluegleam ntpd[565]: Deleting interface #16 xvif4i0, 
} fe80::216:3eff:fe01:705%12#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=259211 secs
} Feb 25 20:37:10 bluegleam ntpd[565]: Deleting interface #17 xvif4i0, 
} 2001:470:de9f:1:216:3eff:fe01:705#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=259211 secs
} Feb 25 20:37:12 bluegleam /netbsd: [ 259252.6908885] xbd backend: attach 
} device vm-www (size 20971520) for domain 29
} Feb 25 20:37:12 bluegleam /netbsd: [ 259252.9408814] xvif29i0: Ethernet 
} address 00:16:3e:01:07:05
} Feb 25 20:37:12 bluegleam /netbsd: [ 259253.7508729] xbd backend domain 29 
} handle 0x300 (768) using event channel 62, protocol x86_64-abi
} Feb 25 20:37:13 bluegleam ntpd[565]: Listen normally on 31 xvif29i0 
} [fe80::216:3eff:fe01:705%23]:123
} Feb 25 20:37:20 bluegleam /netbsd: [ 259261.4907822] in6_ifadd: 
} 2001:470:de9f:1:ec4:7aff:fe03:610 is already configured
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 32 xvif24i0 
} [2001:470:de9f:1:216:3eff:fe01:703]:123
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 33 xvif25i0 
} [2001:470:de9f:1:216:3eff:fe01:701]:123
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 34 xvif26i0 
} [2001:470:de9f:1:216:3eff:fe01:702]:123
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 35 xvif27i0 
} [2001:470:de9f:1:216:3eff:fe01:706]:123
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 36 xvif28i0 
} [2001:470:de9f:1:216:3eff:fe01:704]:123
} Feb 25 20:37:23 bluegleam ntpd[565]: Listen normally on 37 xvif29i0 
} [2001:470:de9f:1:216:3eff:fe01:705]:123
} Feb 25 20:37:56 bluegleam /netbsd: [ 259297.6503468] xbd backend: detach 
} device internal1/test- for domain 27
} Feb 25 20:37:56 bluegleam /netbsd: [ 259297.7503457] xvif27i0: disconnecting
} Feb 25 20:37:56 bluegleam dhcpcd[219]: if_ifa: if_addrflags6: Device not 
} configured
} Feb 25 20:38:16 bluegleam syslogd[285]: last message repeated 3 times
} Feb 25 20:38:16 bluegleam ntpd[565]: Deleting interface #28 xvif27i0, 
} fe80::216:3eff:fe01:706%20#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=186 secs
} Feb 25 20:38:16 bluegleam ntpd[565]: Deleting interface #35 xvif27i0, 
} 2001:470:de9f:1:216:3eff:fe01:706#123, interface stats: received=0, sent=0, 
} dropped=0, active_time=53 secs
} Feb 25 20:38:49 bluegleam /netbsd: [ 259350.6097125] nfs server data:/data: 
} not responding
>-- End of excerpt from =?UTF-8?B?U3RhZmZhbiBUaG9tw6lu?=