Re: domcreate_launch_dm: unable to add disk devices

[smj%ma.sdf.org@localhost]

> You will have to find if the block script is called, and if so why it is
> failing.

I saw Eric's message in the archives.  This was actually a clean install on
7.0 (and I can't even get 33 to build cleanly from pkgsrc anyway).  I initially
tried 45 from pkgsrc and tried going backwards from there.  The earlier 
versions of 4x do have xend, so I was trying to use that, but I am only using
xencommons at this time.  Again, everything seems fine, I can load up guests 
and they run for about a day or so, but this is now the 3rd or 4th day that 
it has gotten into this state where it cannot create a new domu and the 
existing domus are hung.  

Last few lines of xl dmesg:

(XEN) ioapic_guest_write: apic=0, pin=4, irq=4
(XEN) ioapic_guest_write: new_entry=000100f1
(XEN) ioapic_guest_write: old_entry=000000f1 pirq=-1
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) hap.c:274: d66 failed to allocate from HAP pool
(XEN) hap.c:274: d67 failed to allocate from HAP pool
(XEN) hap.c:274: d68 failed to allocate from HAP pool
(XEN) hap.c:274: d69 failed to allocate from HAP pool

I did have a set -x in /usr/pkg/etc/xen/scripts/block and while things were 
fine after a reboot I would get -x output from the script, but when
they are hung like they are now it seems that it never gets far enough to 
call block.  I cannot stop/restart xencommons either, though a kill -9 will
take down xenstored and xenconsoled but ksh /etc/rc.d/xencommons start does
not correct anything, only a reboot does that.

xentop - 15:10:38   Xen 4.5.2
4 domains: 1 running, 3 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 25156664k total, 4534096k used, 20622568k free    CPUs: 16 @ 3200MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) 
  debian16 --b---        295    0.6    1048576    4.2    1049600       4.2
  Domain-0 -----r      44275  100.0    1048576    4.2   no limit       n/a
  netbsd49 --b---         98    0.2    1048576    4.2    1049600       4.2
 openbsd11 --b---        177    1.2    1048568    4.2    1049600       4.2

Nothing looks strange here, xl shutdown/destroy on the domus works, but 
creating them again just produces:

libxl: error: libxl_create.c:1153:domcreate_launch_dm: unable to add disk devices

/var/log/xen/xenstored.log lines pretty much only look like this:

Got message READ len 22 from 0x7f7ff7b39d90
Writing msg ERROR (ENOENT) out to 0x7f7ff7b39d90
OUT 0x7f7ff7b39d90 20160430 15:12:59 ERROR (ENOENT )
IN 0x7f7ff7b39d90 20160430 15:12:59 READ (/local/domain/59/name )
Got message READ len 22 from 0x7f7ff7b39d90
Writing msg ERROR (ENOENT) out to 0x7f7ff7b39d90
OUT 0x7f7ff7b39d90 20160430 15:12:59 ERROR (ENOENT )
IN 0x7f7ff7b39d90 20160430 15:12:59 READ (/local/domain/61/name )
Got message READ len 22 from 0x7f7ff7b39d90
Writing msg READ (openbsd11) out to 0x7f7ff7b39d90
OUT 0x7f7ff7b39d90 20160430 15:12:59 READ (openbsd11)

A reboot fixes it, though that sometimes requires a reboot 0c from the 
debugger.

Which looks like this:

^@++++fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip ffffffff80206465 cs e030 rflags 202 cr2 7f7ff5f29f80 ilevel 8 rsp ffffa000302307e8
curlwp 0xffffa0000542c340 pid 0.57 lowest kstack 0xffffa0003022e2c0
Stopped in pid 0.57 (system) at netbsd:breakpoint+0x5:  leave
breakpoint() at netbsd:breakpoint+0x5
xencons_tty_input() at netbsd:xencons_tty_input+0xb2
xencons_intr() at netbsd:xencons_intr+0x4c
evtchn_do_event() at netbsd:evtchn_do_event+0x281
do_hypervisor_callback() at netbsd:do_hypervisor_callback+0x147
hypervisor_callback() at netbsd:hypervisor_callback+0x9e
vrevoke() at netbsd:vrevoke+0x93
genfs_revoke() at netbsd:genfs_revoke+0xd
VOP_REVOKE() at netbsd:VOP_REVOKE+0x53
vdevgone() at netbsd:vdevgone+0x4f
ldenddetach() at netbsd:ldenddetach+0x98
ld_twa_detach() at netbsd:ld_twa_detach+0x2e
config_detach() at netbsd:config_detach+0x8b
twa_shutdown() at netbsd:twa_shutdown+0x65
doshutdownhooks() at netbsd:doshutdownhooks+0x31
cpu_reboot() at netbsd:cpu_reboot+0x1d
vpanic() at netbsd:vpanic+0x145
snprintf() at netbsd:snprintf
startlwp() at netbsd:startlwp
alltraps() at netbsd:alltraps+0x9f
ds          18c0
es          8d8
fs          c340
gs          8d8
rdi         ffffa00001fa0198
rsi         ffffffff80e15ab0    rbuf.11859
rbp         ffffa000302307e8
rbx         ffffffff80e15ab0    rbuf.11859
rdx         2b
rcx         2b
rax         1
r8          4
r9          ffffa000054661d0
r10         ffffa000302309d0
r11         e033
r12         ffffa00001fca940
r13         ffffffff80e15ab1    rbuf.11859+0x1
r14         ffffa00001fa0198
r15         ffffa00001fc1fc0
rip         ffffffff80206465    breakpoint+0x5
cs          e030
rflags      202
rsp         ffffa000302307e8
ss          e02b
netbsd:breakpoint+0x5:  leave
db>