Netbsd XEN nothing I can boot in

[Ivan Vari <i.vari%niwa.co.nz@localhost>]

Hi all,

I have been trying to use NetBSD as guest for 3 weeks and I can't get it 
to boot at all. This mailing list is my last chance I suppose before we 
stop moving toward NetBSD as a base for our new production systems. I 
don't want Linux for production so pls help! I try to provide as much as 
I can so this email may become long...my apologies.

Machine:
HP Proliant DL380 G3, 2xXeon 3Ghz(+HT), 4GB Ram, 200GB disk space in 
raid5 via Smartarray 5i controller.

Brief history:
stage1: tried NetBSD 3.1RC1 as dom0. System installed ok (generic 
kernel) however the dom0 (kernel) was crashed and got frozen as soon as 
the boot reached the cciss (smartarray 5i driver) driver stage. I 
couldn't do much about it even with my own kernel. (re-compiled dom0 
kernel-same freezing at same stage) I blamed the cciss driver?!

stage2: tried Debian Etch as dom0. System installed ok, dom0 loaded ok. 
LVM was created for guests. I tried at least 15 different kernels 
including "INSTALL" ones, daily snapshot ones, from all versions above 
3.0. None of them was able to boot in. I thought it's Debian's issue 
even though I re-compiled my kernel again and installed Xen 3.0.2 from 
source.

stage3: at present I am trying it with SuSE Openenterprise server as Xen 
3.0.2 dom0. Same problem. same effect an I am tired of it. I have read 
awful lot of information and didn't find much relating my problem with 
google. I have always checked the kernel candidates with
objdump -s -j __xen_guest <kernelname>
and all of them was Xen3 kernel. Just to confirm my theory I tried 
Debian,SuSE even FreeBSD7 test kernels and they all load in OK! I don't 
get console or anything for NetBSD ones, my domain crashes in 2 sec 
according to the log file.

Here it is one of my config files but I have at least 15 different one:
 kernel = "/tmp/netbsd-INSTALL_XEN3_DOMU"
 memory = 256
 name = "NetBSD"
 disk = [ 'file:/tmp/i386cd-3.1_RC2.iso,cd0a,r', 
'phy:mapper/Localraid5-VMPOOL1,0x301,w' ]
 root = "/dev/cd0a"

I tried changing the device labelling schema (also not using LVM) but it 
made no affect (it perhaps does if I loaded the kernel in and wanted 
installation)

I have a bad feeling that I keep doing something wrong which is causing 
this. I disabled HT and 1 CPU but they made no affect, my kernel is 
still SMP though. Any help would be much appreciated.

Ivan

xen:~ # xm create -c /etc/xen/vm/vm2
Using config file "/etc/xen/vm/vm2".
Started domain NetBSD
xen:~ # xm list

Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0     3609     2 r-----   226.6


Xend.log:
[2006-09-13 12:30:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:180) 
XendDomainInfo.create(['vm', ['name', 'NetBSD'], ['memory', 256], 
['root', '/dev/cd0a'], ['vcpus', 1], ['root', '/dev/cd0a'], ['image', 
['linux', ['kernel', '/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', 
'/dev/cd0a']]], ['device', ['vbd', ['uname', 
'file:/tmp/i386cd-3.1_RC2.iso'], ['dev', 'cd0a'], ['mode', 'r']]], 
['device', ['vbd', ['uname', 'phy:mapper/Localraid5-VMPOOL1'], ['dev', 
'0x301'], ['mode', 'w']]], ['device', ['vif', ['mac', 
'ae:00:00:78:bd:00']]]])
[2006-09-13 12:30:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:286) 
parseConfig: config is ['vm', ['name', 'NetBSD'], ['memory', 256], 
['root', '/dev/cd0a'], ['vcpus', 1], ['root', '/dev/cd0a'], ['image', 
['linux', ['kernel', '/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', 
'/dev/cd0a']]], ['device', ['vbd', ['uname', 
'file:/tmp/i386cd-3.1_RC2.iso'], ['dev', 'cd0a'], ['mode', 'r']]], 
['device', ['vbd', ['uname', 'phy:mapper/Localraid5-VMPOOL1'], ['dev', 
'0x301'], ['mode', 'w']]], ['device', ['vif', ['mac', 
'ae:00:00:78:bd:00']]]]
[2006-09-13 12:30:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:382) 
parseConfig: result is {'ssidref': None, 'uuid': None, 'on_crash': None, 
'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', 
'/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', '/dev/cd0a']], 'on_poweroff': 
None, 'cpus': None, 'name': 'NetBSD', 'backend': [], 'cpu': None, 
'vcpus': 1, 'cpu_weight': None, 'vcpu_avail': None, 'memory': 256, 
'device': [('vbd', ['vbd', ['uname', 'file:/tmp/i386cd-3.1_RC2.iso'], 
['dev', 'cd0a'], ['mode', 'r']]), ('vbd', ['vbd', ['uname', 
'phy:mapper/Localraid5-VMPOOL1'], ['dev', '0x301'], ['mode', 'w']]), 
('vif', ['vif', ['mac', 'ae:00:00:78:bd:00']])], 'bootloader': None, 
'root': '/dev/cd0a', 'bootentry': None, 'maxmem': None}
[2006-09-13 12:30:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:1189) 
XendDomainInfo.construct: None 0
[2006-09-13 12:30:16 xend.XendDomainInfo] DEBUG (XendDomainInfo:1221) 
XendDomainInfo.initDomain: 11 1.0
[2006-09-13 12:30:16 xend] DEBUG (balloon:137) Balloon: 262244 KiB free; 
need 262144; done.
[2006-09-13 12:30:16 xend] INFO (image:136) buildDomain os=linux dom=11 
vcpus=1
[2006-09-13 12:30:16 xend] DEBUG (image:176) dom            = 11
[2006-09-13 12:30:16 xend] DEBUG (image:177) image          = 
/tmp/netbsd-INSTALL_XEN3_DOMU
[2006-09-13 12:30:16 xend] DEBUG (image:178) store_evtchn   = 1
[2006-09-13 12:30:16 xend] DEBUG (image:179) console_evtchn = 2
[2006-09-13 12:30:16 xend] DEBUG (image:180) cmdline        = 
root=/dev/cd0a
[2006-09-13 12:30:16 xend] DEBUG (image:181) ramdisk        =
[2006-09-13 12:30:16 xend] DEBUG (image:182) vcpus          = 1
[2006-09-13 12:30:16 xend] DEBUG (image:183) features       =
[2006-09-13 12:30:16 xend] DEBUG (blkif:24) exception looking up device 
number for cd0a: [Errno 2] No such file or directory: '/dev/cd0a'
[2006-09-13 12:30:16 xend] DEBUG (DevController:103) DevController: 
writing {'virtual-device': '52490', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/11/52490'} to 
/local/domain/11/device/vbd/52490.
[2006-09-13 12:30:16 xend] DEBUG (DevController:105) DevController: 
writing {'domain': 'NetBSD', 'frontend': 
'/local/domain/11/device/vbd/52490', 'dev': 'cd0a', 'state': '1', 
'params': '/tmp/i386cd-3.1_RC2.iso', 'mode': 'r', 'frontend-id': '11', 
'type': 'file'} to /local/domain/0/backend/vbd/11/52490.
[2006-09-13 12:30:17 xend] DEBUG (blkif:24) exception looking up device 
number for 0x301: [Errno 2] No such file or directory: '/dev/0x301'
[2006-09-13 12:30:17 xend] DEBUG (DevController:103) DevController: 
writing {'virtual-device': '769', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/11/769'} to 
/local/domain/11/device/vbd/769.
[2006-09-13 12:30:17 xend] DEBUG (DevController:105) DevController: 
writing {'domain': 'NetBSD', 'frontend': 
'/local/domain/11/device/vbd/769', 'dev': '0x301', 'state': '1', 
'params': 'mapper/Localraid5-VMPOOL1', 'mode': 'w', 'frontend-id': '11', 
'type': 'phy'} to /local/domain/0/backend/vbd/11/769.
[2006-09-13 12:30:17 xend] DEBUG (DevController:103) DevController: 
writing {'backend-id': '0', 'mac': 'ae:00:00:78:bd:00', 'handle': '0', 
'state': '1', 'backend': '/local/domain/0/backend/vif/11/0'} to 
/local/domain/11/device/vif/0.
[2006-09-13 12:30:17 xend] DEBUG (DevController:105) DevController: 
writing {'mac': 'ae:00:00:78:bd:00', 'state': '1', 'handle': '0', 
'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '11', 'domain': 
'NetBSD', 'frontend': '/local/domain/11/device/vif/0'} to 
/local/domain/0/backend/vif/11/0.
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:675) 
Storing VM details: {'ssidref': '0', 'uuid': 
'74b11022-3d45-fab2-c5ac-1e957afbac18', 'on_reboot': 'restart', 
'start_time': '1158107417.02', 'on_poweroff': 'destroy', 'name': 
'NetBSD', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '256', 'on_crash': 
'restart', 'image': '(linux (kernel /tmp/netbsd-INSTALL_XEN3_DOMU) (root 
/dev/cd0a))', 'maxmem': '256'}
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:700) 
Storing domain details: {'console/ring-ref': '987515', 'console/port': 
'2', 'name': 'NetBSD', 'console/limit': '1048576', 'vm': 
'/vm/74b11022-3d45-fab2-c5ac-1e957afbac18', 'domid': '11', 
'cpu/0/availability': 'online', 'memory/target': '262144', 
'store/ring-ref': '987516', 'store/port': '1'}
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
vif.
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:881) 
XendDomainInfo.handleShutdownWatch
[2006-09-13 12:30:17 xend] DEBUG (DevController:138) Waiting for 0.
[2006-09-13 12:30:17 xend] DEBUG (DevController:403) 
hotplugStatusCallback /local/domain/0/backend/vif/11/0/hotplug-status.
[2006-09-13 12:30:17 xend] DEBUG (DevController:403) 
hotplugStatusCallback /local/domain/0/backend/vif/11/0/hotplug-status.
[2006-09-13 12:30:17 xend] DEBUG (DevController:417) 
hotplugStatusCallback 1.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
usb.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
vbd.
[2006-09-13 12:30:17 xend] DEBUG (DevController:138) Waiting for 52490.
[2006-09-13 12:30:17 xend] DEBUG (DevController:403) 
hotplugStatusCallback /local/domain/0/backend/vbd/11/52490/hotplug-status.
[2006-09-13 12:30:17 xend] DEBUG (DevController:403) 
hotplugStatusCallback /local/domain/0/backend/vbd/11/52490/hotplug-status.
[2006-09-13 12:30:17 xend] DEBUG (DevController:417) 
hotplugStatusCallback 1.
[2006-09-13 12:30:17 xend] DEBUG (DevController:138) Waiting for 769.
[2006-09-13 12:30:17 xend] DEBUG (DevController:403) 
hotplugStatusCallback /local/domain/0/backend/vbd/11/769/hotplug-status.
[2006-09-13 12:30:17 xend] DEBUG (DevController:417) 
hotplugStatusCallback 1.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
irq.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
pci.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
ioports.
[2006-09-13 12:30:17 xend] DEBUG (DevController:132) Waiting for devices 
vtpm.
[2006-09-13 12:30:17 xend] INFO (XendDomain:363) Domain NetBSD (11) 
unpaused.
[2006-09-13 12:30:17 xend.XendDomainInfo] WARNING (XendDomainInfo:823) 
Domain has crashed: name=NetBSD id=11.
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:1364) 
XendDomainInfo.destroyDomain(11)
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:180) 
XendDomainInfo.create(['domain', ['domid', 11], ['uuid', 
'74b11022-3d45-fab2-c5ac-1e957afbac18'], ['ssidref', 0], ['vcpus', 1], 
['vcpu_avail', 1], ['cpu_weight', 1.0], ['memory', 256], ['maxmem', 
256], ['root', '/dev/cd0a'], ['name', 'NetBSD'], ['on_poweroff', 
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', 
['linux', ['kernel', '/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', 
'/dev/cd0a']]], ['device', ['vif', ['backend', 0], ['script', 
'vif-bridge'], ['mac', 'ae:00:00:78:bd:00']]], ['device', ['vbd', 
['backend', 0], ['dev', 'cd0a'], ['uname', 
'file:/tmp/i386cd-3.1_RC2.iso'], ['mode', 'r']]], ['device', ['vbd', 
['backend', 0], ['dev', '0x301'], ['uname', 
'phy:mapper/Localraid5-VMPOOL1'], ['mode', 'w']]], ['state', '----c-'], 
['shutdown_reason', 'crash'], ['cpu_time', 0.000464871], 
['online_vcpus', 1], ['up_time', '0.554826021194'], ['start_time', 
'1158107417.02'], ['store_mfn', 987516], ['console_mfn', 987515]])
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:286) 
parseConfig: config is ['domain', ['domid', 11], ['uuid', 
'74b11022-3d45-fab2-c5ac-1e957afbac18'], ['ssidref', 0], ['vcpus', 1], 
['vcpu_avail', 1], ['cpu_weight', 1.0], ['memory', 256], ['maxmem', 
256], ['root', '/dev/cd0a'], ['name', 'NetBSD'], ['on_poweroff', 
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', 
['linux', ['kernel', '/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', 
'/dev/cd0a']]], ['device', ['vif', ['backend', 0], ['script', 
'vif-bridge'], ['mac', 'ae:00:00:78:bd:00']]], ['device', ['vbd', 
['backend', 0], ['dev', 'cd0a'], ['uname', 
'file:/tmp/i386cd-3.1_RC2.iso'], ['mode', 'r']]], ['device', ['vbd', 
['backend', 0], ['dev', '0x301'], ['uname', 
'phy:mapper/Localraid5-VMPOOL1'], ['mode', 'w']]], ['state', '----c-'], 
['shutdown_reason', 'crash'], ['cpu_time', 0.000464871], 
['online_vcpus', 1], ['up_time', '0.554826021194'], ['start_time', 
'1158107417.02'], ['store_mfn', 987516], ['console_mfn', 987515]]
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:382) 
parseConfig: result is {'ssidref': 0, 'uuid': 
'74b11022-3d45-fab2-c5ac-1e957afbac18', 'on_crash': 'restart', 
'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['kernel', 
'/tmp/netbsd-INSTALL_XEN3_DOMU'], ['root', '/dev/cd0a']], 'on_poweroff': 
'destroy', 'cpus': None, 'name': 'NetBSD', 'backend': [], 'cpu': None, 
'vcpus': 1, 'cpu_weight': 1.0, 'vcpu_avail': 1, 'memory': 256, 'device': 
[('vif', ['vif', ['backend', 0], ['script', 'vif-bridge'], ['mac', 
'ae:00:00:78:bd:00']]), ('vbd', ['vbd', ['backend', 0], ['dev', 'cd0a'], 
['uname', 'file:/tmp/i386cd-3.1_RC2.iso'], ['mode', 'r']]), ('vbd', 
['vbd', ['backend', 0], ['dev', '0x301'], ['uname', 
'phy:mapper/Localraid5-VMPOOL1'], ['mode', 'w']])], 'bootloader': None, 
'root': '/dev/cd0a', 'bootentry': None, 'maxmem': 256}
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:1189) 
XendDomainInfo.construct: None 0
[2006-09-13 12:30:17 xend.XendDomainInfo] DEBUG (XendDomainInfo:1221) 
XendDomainInfo.initDomain: 12 1.0
[2006-09-13 12:30:17 xend] DEBUG (balloon:143) Balloon: 51136 KiB free; 
211076 to scrub; need 262144; retries: 10.
[2006-09-13 12:30:17 xend] DEBUG (balloon:151) Balloon: waiting on scrubbing
[2006-09-13 12:30:18 xend] DEBUG (balloon:137) Balloon: 262244 KiB free; 
need 262144; done.
[2006-09-13 12:30:18 xend] INFO (image:136) buildDomain os=linux dom=12 
vcpus=1
[2006-09-13 12:30:18 xend] DEBUG (image:176) dom            = 12
[2006-09-13 12:30:18 xend] DEBUG (image:177) image          = 
/tmp/netbsd-INSTALL_XEN3_DOMU
[2006-09-13 12:30:18 xend] DEBUG (image:178) store_evtchn   = 1
[2006-09-13 12:30:18 xend] DEBUG (image:179) console_evtchn = 2
[2006-09-13 12:30:18 xend] DEBUG (image:180) cmdline        = 
root=/dev/cd0a
[2006-09-13 12:30:18 xend] DEBUG (image:181) ramdisk        =
[2006-09-13 12:30:18 xend] DEBUG (image:182) vcpus          = 1
[2006-09-13 12:30:18 xend] DEBUG (image:183) features       =
[2006-09-13 12:30:18 xend] DEBUG (DevController:103) DevController: 
writing {'backend-id': '0', 'mac': 'ae:00:00:78:bd:00', 'handle': '0', 
'state': '1', 'backend': '/local/domain/0/backend/vif/12/0'} to 
/local/domain/12/device/vif/0.
[2006-09-13 12:30:18 xend] DEBUG (DevController:105) DevController: 
writing {'mac': 'ae:00:00:78:bd:00', 'state': '1', 'handle': '0', 
'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '12', 'domain': 
'NetBSD', 'frontend': '/local/domain/12/device/vif/0'} to 
/local/domain/0/backend/vif/12/0.
[2006-09-13 12:30:18 xend] DEBUG (blkif:24) exception looking up device 
number for cd0a: [Errno 2] No such file or directory: '/dev/cd0a'
[2006-09-13 12:30:18 xend] DEBUG (DevController:103) DevController: 
writing {'virtual-device': '52490', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/12/52490'} to 
/local/domain/12/device/vbd/52490.
[2006-09-13 12:30:18 xend] DEBUG (DevController:105) DevController: 
writing {'domain': 'NetBSD', 'frontend': 
'/local/domain/12/device/vbd/52490', 'dev': 'cd0a', 'state': '1', 
'params': '/tmp/i386cd-3.1_RC2.iso', 'mode': 'r', 'frontend-id': '12', 
'type': 'file'} to /local/domain/0/backend/vbd/12/52490.
[2006-09-13 12:30:18 xend] DEBUG (blkif:24) exception looking up device 
number for 0x301: [Errno 2] No such file or directory: '/dev/0x301'
[2006-09-13 12:30:18 xend] DEBUG (DevController:103) DevController: 
writing {'virtual-device': '769', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/12/769'} to 
/local/domain/12/device/vbd/769.
[2006-09-13 12:30:18 xend] DEBUG (DevController:105) DevController: 
writing {'domain': 'NetBSD', 'frontend': 
'/local/domain/12/device/vbd/769', 'dev': '0x301', 'state': '1', 
'params': 'mapper/Localraid5-VMPOOL1', 'mode': 'w', 'frontend-id': '12', 
'type': 'phy'} to /local/domain/0/backend/vbd/12/769.
[2006-09-13 12:30:18 xend.XendDomainInfo] DEBUG (XendDomainInfo:675) 
Storing VM details: {'ssidref': '0', 'uuid': 
'74b11022-3d45-fab2-c5ac-1e957afbac18', 'on_reboot': 'restart', 
'start_time': '1158107418.44', 'on_poweroff': 'destroy', 'name': 
'NetBSD', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '256', 'on_crash': 
'restart', 'image': '(linux (kernel /tmp/netbsd-INSTALL_XEN3_DOMU) (root 
/dev/cd0a))', 'maxmem': '256'}
[2006-09-13 12:30:18 xend.XendDomainInfo] DEBUG (XendDomainInfo:700) 
Storing domain details: {'console/ring-ref': '987517', 'console/port': 
'2', 'name': 'NetBSD', 'console/limit': '1048576', 'vm': 
'/vm/74b11022-3d45-fab2-c5ac-1e957afbac18', 'domid': '12', 
'cpu/0/availability': 'online', 'memory/target': '262144', 
'store/ring-ref': '987518', 'store/port': '1'}
[2006-09-13 12:30:18 xend.XendDomainInfo] WARNING (XendDomainInfo:823) 
Domain has crashed: name=NetBSD id=12.
[2006-09-13 12:30:18 xend.XendDomainInfo] ERROR (XendDomainInfo:1496) VM 
NetBSD restarting too fast (0.904129 seconds since the last restart). 
Refusing to restart to avoid loops.
[2006-09-13 12:30:18 xend.XendDomainInfo] DEBUG (XendDomainInfo:1356) 
XendDomainInfo.destroy: domid=12
[2006-09-13 12:30:18 xend.XendDomainInfo] DEBUG (XendDomainInfo:1364) 
XendDomainInfo.destroyDomain(12)