Need some assistance concerning HVM guests and VLANs

[Hugo Silva <hugo%barafranca.com@localhost>]

Hi list,

Been playing with NetBSD/Xen and a VLAN-aware switch. I'm quite new to 
VLANs, so it's possible I made an elementary error somewhere.

The short description of the problem is: I can't get my FreeBSD HVM 
guest to talk to the network using VLANs.




My switch is an HP ProCurve 1810-G, and I'm using a dedicated OpenBSD 
box to route between VLANs. The OpenBSD port (#1) is on all configured 
VLANs and port #1 is configured as 'T'(agging) for all of them.

The NetBSD dom0 is connected to port 4 and is configured for 'T'agging 
on VLANs 100 and 3. Idea being isolating the hypervisor on VLAN100; this 
is working:

dom0$ ifconfig vlan100
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
       vlan: 100 parent: msk0
       address: --:--:--:--:--:--
       inet 10.100.100.2 netmask 0xffffff00 broadcast 10.100.100.255

There's a NetBSD domU running on VLAN3, which also works just fine:

domU$ ifconfig vlan3
vlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1496
       vlan: 3 parent: xennet0
       address: --:--:--:--:--:--
       inet 10.100.3.4 netmask 0xffffff00 broadcast 10.100.3.255



I'm now trying to convert my FreeBSD HVM guest to this setup;

freebsdHVM$ ifconfig re0.3
re0.3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=3<RXCSUM,TXCSUM>
       ether --:--:--:--:--:--
       inet 10.100.3.3 netmask 0xffffff00 broadcast 10.100.3.255
       media: Ethernet autoselect (100baseTX <full-duplex>)
       status: active
       vlan: 3 parent interface: re0





Everything looks good, I think, but I can't get the FreeBSD guest to 
communicate. Testing connectivity with the others, I try to ping the 
OpenBSD router and notice PF blocking the packet, like so:

21:41:55.395077 rule 0/(match) block in on vlan3: 10.100.3.4 > 
10.100.3.1: icmp: echo request

For this (freebsd) guest, however, I never get to see anything being 
blocked on OpenBSD's side.

If I tcpdump the re0 interface from the HVM guest, I can see the packet 
going out:
21:19:10.512773 --:--:--:--:--:-- > ff:ff:ff:ff:ff:ff, ethertype 802.1Q 
(0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 
10.100.3.1 tell 10.100.3.3, length 28

.. but it is never seen on the OpenBSD side.



The dom0 is configured for bridging;

By the way - I'm not *entirely* sure the problem is the fact that this 
guest is HVM (it could be FreeBSD for all I know), it's an assumption. 
(and we all know what assumptions are :-))

I suspect this is related to tap0, as if I generate traffic to the 
NetBSD domU and tcpdump it's interface (xvif3.0 in this case) from the 
host, I see .1Q packets:

21:25:55.352549 --:--:--:--:--:-- > --:--:--:--:--:--, ethertype 802.1Q 
(0x8100), length 70: vlan 3, p 0, ethertype IPv4, 10.100.2.100.12203 > 
10.100.3.4.22: . ack 4282 win 8303 <nop,nop,timestamp 6629278 2>


But if I do the same for tap0, I notice that I can see the other domU 
(on the same VLAN, 3) provided I'm pinging FreeBSD from NetBSD:

21:32:07.475082 --:--:--:--:--:NETBSD > ff:ff:ff:ff:ff:ff, ethertype 
802.1Q (0x8100), length 68: vlan 3, p 0, ethertype ARP, arp who-has 
10.100.3.3 tell 10.100.3.4
21:32:07.475230 --:--:--:--:--:FREEBSD > --:--:--:--:--:NETBSD, 
ethertype ARP (0x0806), length 42: arp reply 10.100.3.3 is-at 
--:--:--:--:--:FREEBSD

Trying to ping NetBSD from FreeBSD never does anything:
21:34:19.167147 --:--:--:--:--:FREEBSD > ff:ff:ff:ff:ff:ff, ethertype 
ARP (0x0806), length 42: arp who-has 10.100.3.4 tell 10.100.3.3


I think the problem is that for some reason, the .1Q tag is stripped 
"ahead of time" in FreeBSD's case. Not sure, the whole VLAN subject is 
quite new to me.

Thoughts, suggestions, hints?