Re: Xen3 + VLANs + multiple DOM0s cause loss of connectivity?

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Thu, Jan 24, 2008 at 10:11:16PM +0100, Johan Ihren wrote:
> Hi Manuel,
> 
> On 24 Jan 2008, at 15:58, Manuel Bouyer wrote:
> 
> >>Outside of VLANs (i.e. when configuring IPv4 and IPv6 addresses
> >>directly on the xennetN then everything works just fine. VLANs
> >>configured on the DOM0 also works fine. It is just the combination of
> >>Xen3 + DOMU + VLANs that causes problems.
> >
> >What I found strange is that it worked with Xen2. Xen version is  
> >probably
> >not the key here, but the version of dom0.
> 
> >Basically, 802.1q paquets in dom0 are not routed to the bridge  
> >interface but
> >to the vlan interfaces, so these packets can't make it up to the  
> >domUs.
> 
> Umm. There is confusion here, probably mine. I have lots of 802.1q  
> packets that go just fine across the bridge interface between DOMUs in  
> the same DOM0, and they most certainly make it up to the DOMUs. What  
> the packets don't do is go across the physical switch (between DOM0s)  
> that the DOM0 bridge device is connected to. So I have to challenge  
> the assertion that the packets are not routed to the DOM0 bridge  
> interface.

It may depend on which interfaces have vlan(4) attached to. 

> >The way to do this is to have the vlan interfaces in dom0 only,  
> >connect
> >one bridge to each vlan and have in the domU one vif per vlan you  
> >need to
> >connect to.
> 
> Doesn't work for me as I need to be able to dynamically affect  
> topology from inside the DOMUs. I.e. I implement nomadic behaviour by  
> having DOMUs change their VLAN tag. And on occasion I have several  
> dozen VLANs. There's no way I can do that with bridges and bunches of  
> xennets.

Note that you can dynamically create/delete xennet from the dom0 with Xen3.
But it may not help your problem.
I have domUs attached to more than 30 vlans, and it works just fine with
one bridge and one xennet per vlan.

> 
> I remember discussing this with you at a previous occasion when I was  
> trying to have communication between the DOM0(s) and the DOMUs over  
> VLANs (with very limeted success). You explained that the DOM0  
> couldn't do the right thing wrt to both dealing with bridges and vlan  
> interfaces and therefore VLANs on the DOM0 would not see the traffic  
> arriving on the same VLAN from a DOMU (i.e. the bridge gets the  
> packet, not the DOM0 vlan interface). As a consequence of that I  
> stopped using VLANs entirely on the DOM0s and moved all services into  
> yet another DOMU and that has worked just fine for a long time.
> 
> But now, if I understand correctly, you're saying that in the conflict  
> between sending the packet to the VLAN or to the bridge the VLAN gets  
> the packet. That sounds completely contrary to what you said before  
> and not at all in line with my experience.

It's been a time since I looked in details at this code. When I first
set up these domains with lots of network interface, my first idea was
to extend xvif/xennet to properly support 802.1q tagging (i.e. allow packets
4 bytes larger than the ethernet MTU). I looked at vlan and bridge code and
came to the conclusion that it couldn't work, but I don't remember the
details. Especially I don't remember if the vlan would preemt packet from
bridge, or the opposite, or if it would be more random. Also the vlan vs
bridge behavior may have changed between netbsd-3 and netbsd-4, I didn't
check this either.

-- 
Manuel Bouyer, LIP6, Universite Paris VI.           
Manuel.Bouyer%lip6.fr@localhost
     NetBSD: 26 ans d'experience feront toujours la difference
--