At Sun, 26 Jan 2020 12:48:22 -0600, Edgar Pettijohn <edgar%pettijohn-web.com@localhost> wrote:
Subject: can't get guests other than netbsd working
I have tried:
alpine linux
debian linux
freebsd
I have FreeBSD running on a couple of Xen hosts. E.g. this one:
# dmesg
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE r341666 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
VT(vga): text 80x25
XEN: Hypervisor version 4.11 detected.
CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz (3158.87-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x10676 Family=0x6 Model=0x17 Stepping=6
Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,HV>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Hypervisor: Origin = "XenVMMXenVMM"
real memory = 4190109696 (3996 MB)
avail memory = 4017954816 (3831 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
Launching APs: 3 2 1
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810f9770, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc200-0xc20f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
vgapci0: Boot video device
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: console (115200,n,8,1)
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
Timecounters tick every 10.000 msec
usb_needs_explore_all: no devclass
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
xn0: xbd0: <Virtual Network Interface>40960MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: at device/vif/0attaching as ada0
xbd0: on xenbusb_front0
xn0: Ethernet address: 00:16:3e:06:d0:96
features: flush
xbd0: synchronize cache commands enabled.
xenbusb_back0: <Xen Backend Devices>xbd1: on xenstore0
8192MB <Virtual Block Device> at device/vbd/832 on xenbusb_front0
xbd1: xenballoon0: attaching as ada1
<Xen Balloon Device>xbd1: features: flush
xbd1: on xenstore0
synchronize cache commands enabled.
xn0: backend features:
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
WARNING: / was not properly dismounted
lo0: link state changed to UP
xn0: 2 link states coalesced
xn0: link state changed to UP
intsmb0: <Intel PIIX4 SMBUS Interface> irq 20 at device 1.3 on pci0
intsmb0: Could not allocate I/O space
device_attach: intsmb0 attach returned 6
I did the install with their new "FreeBSD*-amd64-raw.xz" install image
image copied into an LVM partition. I've also done the install with the
"FreeBSD*-bootonly.iso" image mounted as an emulated CD-ROM.
The dom0 is a NetBSD 8.99.32 on a Dell PE-2950 with xen{tools,kernel}411
installed. I've also successfully used Xen-4.5 and Xen-4.8 with FreeBSD.
Here are my notes:
Steps to a FreeBSD domU:
=======================
- Must have the line 'type="hvm"' in the domU config
The FreeBSD Handbook says: "Support for para-virtualized (PV)
domains has been removed from FreeBSD 11 in favor of hardware
virtualized (HVM) domains, which provides better performance."
Which is only true because they now support "PV on HVM" in their
GENERIC kernel, i.e. full hardware CPU virtualization (HVM) with
(some) para-virtualized (PV) device drivers (e.g. network and disk)
for superior I/O performance.
- initial configuration with 'serial="pty"' can be specified, but it
uses VNC anyway and a serial console won't work until
/boot/loader.conf in the domU is updated as shown below (unless the
initial filesystem image is already configured that way)
- disks are named "hda", "hdb", "hdd", etc.
# lvm lvcreate -L 40G -n ${domu_name}.0 vg0
# xzcat FreeBSD-12.0-RELEASE-amd64.raw.xz > /dev/mapper/vg0-${domu_name}.0
disk spec is:
'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-${domu_name}.0'
- swap may not be necessary, but may be useful! (also for other disks)
lvm lvcreate -L 8G -n ${domu_name}.1 vg0
disk spec is:
'format=raw, vdev=hdb, access=rw, target=/dev/mapper/vg0-${domu_name}.1'
- with the "raw" disk image as above you must specify 'boot="c"'
(i.e. boot with an emulated hard disk). To boot an ISO mounted as
an emulated CD-ROM, you'll need 'boot="d"' and a disk spec like:
'format=raw, vdev=hdd, access=ro, devtype=cdrom, target=/images/FreeBSD-11.0-RELEASE-amd64-bootonly.iso'
- boot using vncviewer, i.e. "xl create -V ${domu_name}", and then
login as root and run these commands:
echo 'console="comconsole"' >> /boot/loader.conf
echo 'comconsole_speed="115200"' >> /boot/loader.conf
# possibly: vi /etc/rc.conf # to change hostname
# possibly configure swap, other disks, etc.
echo 'growfs_enable=YES' >> /etc/rc.conf
echo 'sshd_enable=YES' >> /etc/rc.conf
halt -pq
to reboot the domU, then "xl create /etc/xen/${domu_name}" in dom0
again to boot with the new console setting (and maybe run
"/etc/rc.d/conserver reload" too to configure its console port(s))
- WARNING: FreeBSD PVHVM abuses QEMU for UART emulation for the
serial console instead of using the PV Xen console support, thus
you'll end up with two PTYs and two conserver connections dedicated
to every FreeBSD PHHVM domU. The one with the "-cons" suffix would
be the Xen console one, but it will be dead and unattached on the
FreeBSD side. (Rumour has it that upcoming "PVH" support will
return to using the Xen console. [PVH requires an Intel chip with
EPT support, and Xen >4.10] https://wiki.xen.org/wiki/FreeBSD_PVH
and https://wiki.xenproject.org/wiki/PVH_(v2)_Domu)
- you'll want to enable "pkg" support (run "pkg"), etc., install
things like "avahi" (for MDNS), "rsync", etc. Watch out for things
using gtk3 though, such as emacs26 -- they now install wayland too!
Gak!
Note I use conserver for managing domU "serial" (pty) consoles.
Here's an example configuration file -- the one for the host shown above:
#
# xl.cfg(5) - startup for a PV-on-HVM FreeBSD guest
#
name = "freebsd"
uuid = "46118b1d-53fb-11e9-ba99-d4ae528b9511"
#
# n.b.: before xentools-4.11 this would be 'builder = "hvm"'
type = "hvm"
#
memory = 1000
maxmem = 4000
vcpus = 4
boot = "c"
vif = [ 'bridge=bridge0' ]
disk = [
'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-freebsd.0',
'format=raw, vdev=hdb, access=rw, target=/dev/mapper/vg0-freebsd.1',
]
# XXX to make this work, boot first using vncviewer, and then run:
# echo 'console="comconsole"' >> /boot/loader.conf; halt -pq
serial = "pty"