Strange events leading to "Hotplug scripts not working" and xen death

[Avleen Vig <lists-netbsd%silverwraith.com@localhost>]

I apologise to anyone who is on xen-users or xen-devel. I posted this to
both lists and was told I should talk to the netbsd xen maintainers :-)

Can anyone here help me with this? I really odn't want to have to use
Linux for my Dom0, but I might be forced to at this rate :-(
I think it's a bug in whatever is supposed to execute the vif-bridge
script (but I'm not sure if that is xend, or xenbackend).

System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3 compiled from pkgsrc.

I have Dom0 working just fine. It boots without a problem.
I also have a file-backed Linux DomU which starts. The config for this
is at the end.

I've found a rather.. strange series of events that causes Xen to stop
creating new DomUs. Even my working Linux DomUs fail to create.
The thing that makes this strange, is that there doesn't seem to be one
single thing thing in particular that causes this. For instance,
following the instructions here on creating a FreeBSD DomU work the
first time, and after that everything stops working:
  
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/virtualization-guest.html

Also, downloading CentOS 5 from jailtime.org causes everything to stop
working. The configs I used for FreeBSD and CentOS5 are at the end also.

When things "stop working", I see messages like this when trying to
create a domain:
  [root@xen1] /xen/configs # xm create -c freebsd1
  Using config file "./freebsd1".
  Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

Some digging indicated that the scripts (eg,
/usr/pkg/etc/xen/scripts/vif-bridge) are not actually being accessed any
more - their access time does not change again until I reboot to fix
this problem. Restarting xend or any other xen daemons doesn't work.

*Finally* :-) commands like 'xenstore-ls' just hang too.
The xenstore commands seem to be hanging trying to read from the store.
/var/run/xenstored/tdb is getting updated though, before 'xm create'
hangs.

I believe the problem may have something to do with the storage, but I'm
not sure.
The end of the xenstore-trace.log looks like this:

OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (447 )
IN  0x805b0b0 20070513 19:53:35 READ (/local/domain/244/device/vif/0/backend )
OUT 0x805b0b0 20070513 19:53:35 READ (/local/domain/0/backend/vif/244/0)
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN  0x805b0b0 20070513 19:53:35 WATCH 
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916 )
CREATE watch 0x8062030
OUT 0x805b0b0 20070513 19:53:35 WATCH (OK )
OUT 0x805b0b0 20070513 19:53:35 WATCH_EVENT 
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916 )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (448 )
IN  0x805b0b0 20070513 19:53:35 READ 
(/local/domain/0/backend/vif/244/0/hotplug-status )
OUT 0x805b0b0 20070513 19:53:35 ERROR (ENOENT )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )


Can anyone help? The only suggestion that the many Google results on
"Hotplug scripts not working" errors say to make sure udev is working,
but I don't think that applies to NetBSD just Linux.


Working Linux DomU config:
  kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
  memory = 128
  name = "Debian_Sarge_Image"
  vif = [ 'mac=00:11:22:33:44:55, bridge=bridge0' ]
  disk = [ 'file:/xen/nfs_root/sarge1/sarge1.img,hda1,w' ]
  root = "/dev/hda1 ro"
  extra = "init=/bin/sh"

FreeBSD 7 which causes breakage:
  kernel = "/xen/kernels/freebsd7-kernel"
  memory = 256
  name = "freebsd_xen_INSTALL"
  vif = [ 'mac=00:11:22:33:44:66, bridge=bridge0' ]
  disk = [ 'file:/xen/nfs_root/freebsd1/mdroot-7.0,hda1,w' ]
  extra = "boot_verbose"
  extra += ",boot_single"
  extra += ",kern.hz=100"
  extra += ",vfs.root.mountfrom=ufs:/dev/xbd769a"

CentOS 5 which causes breakage:
  kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
  memory = 128
  name = "centos.5-0"
  vif = [ '' ]
  dhcp = "dhcp"
  disk = ['file:/xen/nfs_root/centos50/centos.5-0.img,sda1,w', 
'file:/xen/nfs_root/centos50/centos.swap,sda2,w']
  root = "/dev/sda1 ro"

-- 
Avleen Vig
Systems Administrator
Personal: www.silverwraith.com

"It's not me that's crazy, it's you."
"Monkey?"