Re: can't get guests other than netbsd working

[Edgar Pettijohn <edgar%pettijohn-web.com@localhost>]

This should go in the wiki. After reading your notes I see where I was going wrong and what I need to look at next.

Much appreciated.

Edgar
On Jan 27, 2020 2:13 PM, "Greg A. Woods" <woods%planix.ca@localhost> wrote:
>
> At Sun, 26 Jan 2020 12:48:22 -0600, Edgar Pettijohn <edgar%pettijohn-web.com@localhost> wrote:
> Subject: can't get guests other than netbsd working
> > 
> > I have tried:
> > 
> > alpine linux
> > debian linux
> > freebsd
> > 
>
> I have FreeBSD running on a couple of Xen hosts.  E.g. this one:
>
> # dmesg
> ---<<BOOT>>---
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 12.0-RELEASE r341666 GENERIC amd64
> FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
> VT(vga): text 80x25
> XEN: Hypervisor version 4.11 detected.
> CPU: Intel(R) Xeon(R) CPU           X5460  @ 3.16GHz (3158.87-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17  Stepping=6
>   Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
>   Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,HV>
>   AMD Features=0x20100800<SYSCALL,NX,LM>
>   AMD Features2=0x1<LAHF>
> Hypervisor: Origin = "XenVMMXenVMM"
> real memory  = 4190109696 (3996 MB)
> avail memory = 4017954816 (3831 MB)
> Event timer "LAPIC" quality 100
> ACPI APIC Table: <Xen HVM>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s)
> random: unblocking device.
> ioapic0: Changing APIC ID to 1
> MADT: Forcing active-low polarity and level trigger for SCI
> ioapic0 <Version 1.1> irqs 0-47 on motherboard
> Launching APs: 3 2 1
> random: entropy device external interface
> kbd1 at kbdmux0
> netmap: loaded module
> [ath_hal] loaded
> module_register_init: MOD_LOAD (vesa, 0xffffffff810f9770, 0) error 19
> nexus0
> vtvga0: <VT VGA driver> on motherboard
> cryptosoft0: <software crypto> on motherboard
> acpi0: <Xen> on motherboard
> acpi0: Power Button (fixed)
> acpi0: Sleep Button (fixed)
> cpu0: <ACPI CPU> on acpi0
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 62500000 Hz quality 950
> attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> atrtc0: registered as a time-of-day clock, resolution 1.000000s
> Event timer "RTC" frequency 32768 Hz quality 0
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> isab0: <PCI-ISA bridge> at device 1.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc200-0xc20f at device 1.1 on pci0
> ata0: <ATA channel> at channel 0 on atapci0
> ata1: <ATA channel> at channel 1 on atapci0
> pci0: <bridge> at device 1.3 (no driver attached)
> vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
> vgapci0: Boot video device
> xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: model IntelliMouse Explorer, device ID 4
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: does not respond
> device_attach: fdc0 attach returned 6
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
> uart0: console (115200,n,8,1)
> ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
> ppbus0: <Parallel port bus> on ppc0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> xenpv0: <Xen PV bus> on motherboard
> granttable0: <Xen Grant-table Device> on xenpv0
> xen_et0: <Xen PV Clock> on xenpv0
> Event timer "XENTIMER" frequency 1000000000 Hz quality 950
> Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
> xen_et0: registered as a time-of-day clock, resolution 0.000001s
> xenstore0: <XenStore> on xenpv0
> xsd_dev0: <Xenstored user-space device> on xenpv0
> evtchn0: <Xen event channel user-space device> on xenpv0
> privcmd0: <Xen privileged interface user-space device> on xenpv0
> gntdev0: <Xen grant-table user-space device> on xenpv0
> debug0: <Xen debug handler> on xenpv0
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
> Timecounters tick every 10.000 msec
> usb_needs_explore_all: no devclass
> xctrl0: <Xen Control Device> on xenstore0
> xs_dev0: <Xenstore user-space device> on xenstore0
> xenbusb_front0: <Xen Frontend Devices> on xenstore0
> xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
> xn0: xbd0: <Virtual Network Interface>40960MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
> xbd0:  at device/vif/0attaching as ada0
> xbd0:  on xenbusb_front0
> xn0: Ethernet address: 00:16:3e:06:d0:96
> features: flush
> xbd0: synchronize cache commands enabled.
> xenbusb_back0: <Xen Backend Devices>xbd1:  on xenstore0
> 8192MB <Virtual Block Device> at device/vbd/832 on xenbusb_front0
> xbd1: xenballoon0: attaching as ada1
> <Xen Balloon Device>xbd1: features: flush
> xbd1:  on xenstore0
> synchronize cache commands enabled.
> xn0: backend features:
> Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
> WARNING: / was not properly dismounted
> lo0: link state changed to UP
> xn0: 2 link states coalesced
> xn0: link state changed to UP
> intsmb0: <Intel PIIX4 SMBUS Interface> irq 20 at device 1.3 on pci0
> intsmb0: Could not allocate I/O space
> device_attach: intsmb0 attach returned 6
>
> I did the install with their new "FreeBSD*-amd64-raw.xz" install image
> image copied into an LVM partition.  I've also done the install with the
> "FreeBSD*-bootonly.iso" image mounted as an emulated CD-ROM.
>
> The dom0 is a NetBSD 8.99.32 on a Dell PE-2950 with xen{tools,kernel}411
> installed.  I've also successfully used Xen-4.5 and Xen-4.8 with FreeBSD.
>
> Here are my notes:
>
> Steps to a FreeBSD domU:
> =======================
>
>   - Must have the line 'type="hvm"' in the domU config
>
>     The FreeBSD Handbook says:  "Support for para-virtualized (PV)
>     domains has been removed from FreeBSD 11 in favor of hardware
>     virtualized (HVM) domains, which provides better performance."
>
>     Which is only true because they now support "PV on HVM" in their
>     GENERIC kernel, i.e. full hardware CPU virtualization (HVM) with
>     (some) para-virtualized (PV) device drivers (e.g. network and disk)
>     for superior I/O performance.
>
>   - initial configuration with 'serial="pty"' can be specified, but it
>     uses VNC anyway and a serial console won't work until
>     /boot/loader.conf in the domU is updated as shown below (unless the
>     initial filesystem image is already configured that way)
>
>   - disks are named "hda", "hdb", "hdd", etc.
>
> # lvm lvcreate -L 40G -n ${domu_name}.0 vg0
> # xzcat FreeBSD-12.0-RELEASE-amd64.raw.xz > /dev/mapper/vg0-${domu_name}.0
>
>     disk spec is:
>
> 'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-${domu_name}.0'
>
>   - swap may not be necessary, but may be useful!  (also for other disks)
>
> lvm lvcreate -L 8G -n ${domu_name}.1 vg0
>
>     disk spec is:
>
> 'format=raw, vdev=hdb, access=rw, target=/dev/mapper/vg0-${domu_name}.1'
>
>   - with the "raw" disk image as above you must specify 'boot="c"'
>     (i.e. boot with an emulated hard disk).  To boot an ISO mounted as
>     an emulated CD-ROM, you'll need 'boot="d"' and a disk spec like:
>
> 'format=raw, vdev=hdd, access=ro, devtype=cdrom, target=/images/FreeBSD-11.0-RELEASE-amd64-bootonly.iso'
>
>   - boot using vncviewer, i.e.  "xl create -V ${domu_name}", and then
>     login as root and run these commands:
>
>   echo 'console="comconsole"' >> /boot/loader.conf
>   echo 'comconsole_speed="115200"' >> /boot/loader.conf
> # possibly:  vi /etc/rc.conf # to change hostname
> # possibly configure swap, other disks, etc.
> echo 'growfs_enable=YES' >> /etc/rc.conf
> echo 'sshd_enable=YES' >> /etc/rc.conf
> halt -pq
>
>     to reboot the domU, then "xl create /etc/xen/${domu_name}" in dom0
>     again to boot with the new console setting (and maybe run
>     "/etc/rc.d/conserver reload" too to configure its console port(s))
>
>   - WARNING:  FreeBSD PVHVM abuses QEMU for UART emulation for the
>     serial console instead of using the PV Xen console support, thus
>     you'll end up with two PTYs and two conserver connections dedicated
>     to every FreeBSD PHHVM domU.  The one with the "-cons" suffix would
>     be the Xen console one, but it will be dead and unattached on the
>     FreeBSD side.  (Rumour has it that upcoming "PVH" support will
>     return to using the Xen console.  [PVH requires an Intel chip with
>     EPT support, and Xen >4.10] https://wiki.xen.org/wiki/FreeBSD_PVH
>     and https://wiki.xenproject.org/wiki/PVH_(v2)_Domu)
>
>   - you'll want to enable "pkg" support (run "pkg"), etc., install
>     things like "avahi" (for MDNS), "rsync", etc.  Watch out for things
>     using gtk3 though, such as emacs26 -- they now install wayland too!
>     Gak!
>
>
> Note I use conserver for managing domU "serial" (pty) consoles.
>
> Here's an example configuration file -- the one for the host shown above:
>
> #
> # xl.cfg(5) - startup for a PV-on-HVM FreeBSD guest
> #
> name = "freebsd"
> uuid = "46118b1d-53fb-11e9-ba99-d4ae528b9511"
> #
> # n.b.:  before xentools-4.11 this would be 'builder = "hvm"'
> type = "hvm"
> #
> memory = 1000
> maxmem = 4000
> vcpus = 4
> boot = "c"
> vif = [ 'bridge=bridge0' ]
> disk = [
>         'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-freebsd.0',
>         'format=raw, vdev=hdb, access=rw, target=/dev/mapper/vg0-freebsd.1',
> ]
> # XXX to make this work, boot first using vncviewer, and then run:
> # echo 'console="comconsole"' >> /boot/loader.conf; halt -pq
> serial = "pty"
>
> -- 
> Greg A. Woods <gwoods%acm.org@localhost>
>
> Kelowna, BC     +1 250 762-7675           RoboHack <woods%robohack.ca@localhost>
> Planix, Inc. <woods%planix.com@localhost>     Avoncote Farms <woods%avoncote.ca@localhost>