I have a few Xen servers with NetBSD dom0, and have been running
primarily NetBSD domUs, with one production FreeBSD dom0.
I started with, and am still running, Xen 4.5.0nb1 kernel with
xentools-4.5.0nb1 on two machines, while the third has Xen 4.8.
The first FreeBSD was 11.0 I think, and it was installed from an ISO
boot under Xen. It's now upgraded to 11.2, still on Xen 4.8.
I now have FreeBSD-12.0 installed from their new "raw" VM image and
running as shown below.
Both FreeBSD 11 and 12 boot a GENERIC kernel and detect Xen and
configure their PV drivers for HVM mode. I.e. PV-on-HVM or PVHVM.
There is one minor annoyance -- they have the PV xencons driver in the
kernel, but I've been unable to convince the kernel to use it as the
console, or indeed to even attach it at all. As a result if you set up
a 'console="comconsole"' then you end up with two PTYs dedicated to
every FreeBSD VM. Hopefully full PVH mode will fix this. In fact I'm
going to try it this week once I upgrade a machine to Xen-4.11 -- I
think FreeBSD-12.0 is ready for full PVH in GENERIC, though I've yet to
confirm this anywhere, other than what this Xen wiki says:
https://wiki.xen.org/wiki/FreeBSD_PVH
I have not yet done any performance studies, even trivial, but
everything I read says PVH will be the best of both/all Xen worlds.
The future is PVH domUs!
====================
#
# xl.cfg(5) - startup for a PV-on-HVM FreeBSD guest
#
name = "freebsd"
uuid = "46118b1d-53fb-11e9-ba99-d4ae528b9511"
#
# n.b.: in xentools-4.11 this will change to 'type'
builder = "hvm"
#
memory = 1000
maxmem = 4000
vcpus = 4
boot = "c"
vif = [ 'bridge=bridge0' ]
disk = [
# lvm lvcreate -L 40G -n lv40 vg0
# xzcat /images/FreeBSD-12.0-RELEASE-amd64.raw.xz > /dev/mapper/vg0-lv40
# boot single-user and run /etc/rc.d/growfs forcestart
'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-freebsd.0',
# lvm lvcreate -L 8G -n lv41 vg0
'format=raw, vdev=hdb, access=rw, target=/dev/mapper/vg0-freebsd.1',
# or if you want to install (and change 'boot = "d"' to boot from emulated CD-ROM)
# 'format=raw, vdev=hdd, access=ro, devtype=cdrom, target=/images/FreeBSD-11.0-RELEASE-amd64-bootonly.iso'
]
# XXX none of this is necessary -- vnc is the default
#vnc = 1
#vnclisten = '127.0.0.1'
#vncdisplay = 1
# xxx this could be useful...
#vncpassword = 'blah'
#usbdevice = "tablet"
# XXX to make serial console actually work, boot with vncviewer, then:
# echo 'console="comconsole"' >> /boot/loader.conf
serial = "pty"
====================
My more detailed notes are as follows:
(notes mentioning conserver expect my "mk-conserver.sh" script is
installed with Xen to auto-configure conserver from xenstore)
Steps to a FreeBSD domU:
=======================
- Must have the line 'builder="hvm"' in the domU config
The FreeBSD Handbook says: "Support for para-virtualized (PV)
domains has been removed from FreeBSD 11 in favor of hardware
virtualized (HVM) domains, which provides better performance."
Which is only true because they now support "PV on HVM" in their
GENERIC kernel, i.e. full hardware CPU virtualization (HVM) with
(some) para-virtualized (PV) device drivers (e.g. network and disk)
for superior I/O performance.
- initial configuration with 'serial="pty"' can be specified, but it
uses VNC anyway and a serial console won't work until
/boot/loader.conf in the domU is updated as shown below (unless the
initial filesystem image is already configured that way)
- disks are named "hda", "hdb", "hdd", etc.
lvm lvcreate -L 40G -n ${domu_name}.0 vg0
xzcat FreeBSD-12.0-RELEASE-amd64.raw.xz > /dev/mapper/vg0-${domu_name}.0
'format=raw, vdev=hda, access=rw, target=/dev/mapper/vg0-${domu_name}.0'
- swap may not be necessary?
lvm lvcreate -L 8G -n ${domu_name}.1 vg0
- with the raw disk image you can start with 'boot="c"' (i.e. boot
with an emulated hard disk.
- boot using vncviewer, i.e. "xl create -V ${domu_name}", and then
login as root and run these commands:
echo 'console="comconsole"' >> /boot/loader.conf
echo 'comconsole_speed="115200"' >> /boot/loader.conf
# possibly: vi /etc/rc.conf # to change hostname
echo 'growfs_enable=YES' >> /etc/rc.conf
echo 'sshd_enable=YES' >> /etc/rc.conf
halt -pq
to reboot the domU, then "xl create /etc/xen/${domu_name}" in dom0
again to enable the new console setting (and maybe run
"/etc/rc.d/conserver reload" too to configure its console port(s))
- WARNING: FreeBSD PVHVM abuses QEMU for UART emulation for the
serial console instead of using the PV Xen console support, thus
you'll end up with two PTYs and two conserver connections dedicated
to every FreeBSD PHHVM domU. The one with the "-cons" suffix would
be the Xen console one, but it will be dead and unattached on the
FreeBSD side. (Rumour has it that upcoming "PVH" support will
return to using the Xen console. [PVH requires an Intel chip with
EPT support, and Xen >4.10] https://wiki.xen.org/wiki/FreeBSD_PVH
and https://wiki.xenproject.org/wiki/PVH_(v2)_Domu)
- you'll want to enable "pkg" support (run "pkg"), etc., install
things like "avahi" (for MDNS), "rsync", etc. Watch out for things
using gtk3 though, such as emacs26 -- they now install wayland too!
Gak!
--
Greg A. Woods <gwoods%acm.org@localhost>
+1 250 762-7675 RoboHack <woods%robohack.ca@localhost>
Planix, Inc. <woods%planix.com@localhost> Avoncote Farms <woods%avoncote.ca@localhost>
Attachment:
pgpMzg42uBnHV.pgp
Description: OpenPGP Digital Signature