Re: Xen3/PAE problems with domainU

[Mathieu Ropert <mro%adviseo.fr@localhost>]

Manuel Bouyer wrote:
> On Fri, Aug 04, 2006 at 09:00:31PM -0400, Todd Vierling wrote:
>   
> > This is Bcc:'d to port-i386 as it also concerns PAE, which may be required
> > to get things working correctly.  The main thread should remain on port-xen.
> >
> > So I am in contact with the proprietor of a hosting company that is offering
> > Xen-based hosting, and is moving to Xen 3.  We had a pretty well working
> > NetBSD domU setup for Xen2.  But for PAE and other things, this host has to
> > move to Xen3.
> >
> > I got the 20060803 build of the Xen3 kernels, and the host ran into this
> > when creating the domain.  The host has Linux kernel 2.6.16 compiled
> > directly from xensource + distro sources, and has xentools 3.0.2-2
> > installed.
> >
> > ==========
> >
> > [...]
> >
> > ==========
> >
> > I found the following very vague post about a problem just like this when
> > the dom0 has PAE enabled but the domU doesn't:
> >
> >     http://lists.xensource.com/archives/html/xen-users/2006-06/msg00077.html
> >
> > Is there any PAE support in work for i386 (not just amd64)?  Looks like the
> > Xen folks aren't going to emulate non-PAE domU environments on a PAE dom0
> > anytime soon, so this has become a requirement.
> >     
>
> I'm not sure NetBSD/i386 itself will get PAE support, AFAIK nobody is
> working on it. I don't know how much work is involved, or what it
> would mean for userland compatibility (especially kmem grovelers).
>
> I don't know if a Xen/x86_PAE could boot a guest kernel for x86_64
> (you could probably try with a linux x86_64 domU kernel). It it does the way
> to go would be to port NetBSD/amd64 to Xen. There's patches for OpenBSD so
> maybe it's not that much work to get a domU kernel working.
> But I don't have the time to work on this right now (I want to get dom0/i386
> working right first), and some manpower is definitively welcome :)
>
>   
I've done the original work on the OpenBSD/amd64 port for Xen and yes I 
got trouble with the memory management. The way it's currently done in 
BSD is incompatible with Xen till its get patched. I've applied some 
brutal fixes to be able to continue developpement, but it's clear than 
Xen will need further improvements in that direction.
Fortunately, the problem doesn't manifest with 2 levels page tables, so 
i386 is safe. However, it's buggy with 4 level tables (amd64), and 
should probably be with 3 levels too (i386 PAE).
About combining amd64 and i386 with PAE, it should be possible to run an 
i386 domain on an amd64 hypervisor, but not the opposite. And if i'm not 
mistaken, you can run an i386 domain on an amd64 hypervisor only if you 
use VMX (ie hardware virtualisation). The paravirtualized amd64 version 
of Xen is not designed to support a kernel in i386 compatibility mode 
(domains are started in 64 bit environnement).
Paravirtualized i386 and i386/PAE combine won't work either, since a PAE 
hypervisor will expect PAE tables from guests kernels.