Re: XenServer and viridian

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

On Thu, 27 Mar 2025, Manuel Bouyer wrote:
> > > > > > Can you give more details about the panic ?
> > > > >
> > > > > [   1.0000000] NetBSD 10.1_STABLE (GENERIC) #6: Wed Mar 26
> > > > > 17:13:37 GMT 2025
> > > > > [   1.0000000]
> > > > > root@builder10:/usr/work/obj/10/amd64/sys/arch/amd64/compile/GENERIC
> > > > > [   1.0000000] total memory = 1019 MB
> > > > > [   1.0000000] avail memory = 957 MB
> > > > > [   1.0000040] efi: systbl at pa 3f5ee018
> > > > > [   1.0000040] mainbus0 (root)
> > > > > [   1.0000040] Identified Guest XEN in HVM mode.
> > > > > [   1.0000040] fatal protection fault in supervisor mode
> > > > > [   1.0000040] trap type 4 code 0 rip 0xffffffff802499cd cs 0x8
> > > > > rflags 0x246 cr2 0 ilevel 0x8 rsp 0xffffffff81d40cf0
> > > > > [   1.0000040] curlwp 0xffffffff8188b000 pid 0.0 lowest kstack
> > > > > 0xffffffff81d3b2c0
> > > > > kernel: protection fault trap, code=0
> > > > > Stopped in pid 0.0 (system) at  netbsd:xen_hvm_init+0xad:       wrmsr
> > > > > xen_hvm_init() at netbsd:xen_hvm_init+0xad
> > > >
> > > > So my guess is that it's failing in xen_init_hypercall_page(),
> > > > when doing:
> > > >        wrmsr(descs[1], (uintptr_t)&hypercall_page - KERNBASE);
> > > >
> > > > this means that when booting in this mode, the Xen interfaces are not
> > > > available to the guest.
> > > >
> > > > When booting in this mode without the hack, do you see some unattached Xen
> > > > PCI devices in the dmesg ?
> > >
> > > Yes:
> > > # dmesg | grep -i xen
> > > ACPI: RSDP 0x000000003F5FA014 000024 (v02 Xen   )
> > > ACPI: XSDT 0x000000003F5F90E8 00004C (v01 Xen    HVM      00000000
> > > 01000013)
> > > ACPI: FACP 0x000000003F5F8000 0000F4 (v04 Xen    HVM      00000000 HVML
> > > 00000000)
> > > ACPI: DSDT 0x000000003E50C000 0092A3 (v02 Xen    HVM      00000000 INTL
> > > 20160527)
> > > ACPI: APIC 0x000000003F5F7000 000260 (v02 Xen    HVM      00000000 HVML
> > > 00000000)
> > > ACPI: HPET 0x000000003F5F6000 000038 (v01 Xen    HVM      00000000 HVML
> > > 00000000)
> > > ACPI: WAET 0x000000003F5F5000 000028 (v01 Xen    HVM      00000000 HVML
> > > 00000000)
> > > acpi0: X/RSDT: OemId <   Xen,     HVM,00000000>, AslId <    ,01000013>
> > > XenSource, Inc. Xen Platform Device (undefined, subclass 0x80, revision
> > > 0x01) at pci0 dev 3 function 0 not configured
> >
> > I also tried FreeBSD 14.2 with exactly the same config and they get it
> > right.
>
> OK, so in this case the interface to use is proably though the
> "Xen Platform Device" PCI device, as the native Xen interface is not available.
>
> More code needed ...

I don't think that's the case. On FreeBSD xenpci0 attaches to that device, 
but appears to do nothing. If I disable the device at boot time, 
everything still works with or without viridian. Xen is detected 
extremely early on. Abridged dmesg on FreeBSD 14.2 here (viridian = true,
xenpci disabled):

Hyper-V Version: 0.0.0 [SP0]
  Features=0x870<APIC,HYPERCALL,VPINDEX,TMFREQ>
  PM Features=0x0 [C0]
  Features3=0x8<PCPUDPE>
XEN: Hypervisor version 4.13 detected.
[snip]
xenpv0: <Xen PV bus>
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 96:fe:50:f8:b9:e6
xenbusb_back0: <Xen Backend Devices> on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xbd0: 10240MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: write_barrier
xbd0: synchronize cache commands enabled.

I also checked out OpenBSD 7.6 (after a few mis-steps where it panics if 
trying to boot the installer in uEFI mode, so I had to revert to BIOS 
mode). OpenBSD has an interesting model where it explicitly supports
multiple hypervisors (or at least their interfaces) at the same time on.
pvbus(4) is hypervisor-agnostic with a xen driver that attaches when on 
Xen. Note the two hypervisor types listed on pvbus0:

pvbus0 at mainbus0: Hyper-V 0.0, Xen 4.13
xen0 at pvbus0: features 0x2705, 64 grant table frames, event channel 3
xbf0 at xen0 backend 0 channel 8: cdrom
xbf0: timed out waiting for backend to connect
xbf1 at xen0 backend 0 channel 8: disk
scsibus1 at xbf1: 1 targets
sd0 at scsibus1 targ 0 lun 0: <Xen, phy xvda 768, 0000>
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
xnf0 at xen0 backend 0 channel 9: address da:ae:95:33:63:ed

OpenBSD attaches xspd0 to the PCI device:
xspd0 at pci0 dev 3 function 0 "XenSource Platform Device" rev 0x01

But again, this appears to do nothing. If I disable xspd at userconf, 
nothing is affected.

Another nice thing in OpenBSD is that base includes hostctl(8), a generic 
tool for working with key/value stores on the host (supports Hyper-V, Xen 
and VMware). On Xen, this is how xenstore is manipulated (without needing
the equivalent of sysutils/xenstoretools). With viridian=true, this does 
not work as it thinks it is on Hyper-V, but it does work well with 
viridian=false
# hostctl -t
/dev/pvbus0: Hyper-V

As an aside, neither FreeBSD and OpenBSD get confused by the ghost wd0 
at isa0 which plagues NetBSD. FreeBSD also live migrates. OpenBSD fails, 
like NetBSD in this regard.

Again, I can provide HTML5 console access to any/all of these VMs if 
desired.

--
Stephen