Re: tcpdump on tap100: no packets shown, even though they arrive/get replied to

[Hugo Silva <hugo%barafranca.com@localhost>]

On 08/24/11 18:11, Robert Swindells wrote:
> Hugo Silva wrote:
>> On 08/24/11 16:55, Robert Swindells wrote:
>>> Hugo Silva wrote:
>>>> Just noticed this;
>>>>
>>>> 15:18:05.215970 00:16:3e:be:ef:03 f2:0b:a4:63:80:03 0800 98:
>>>> 172.16.99.203 > 172.16.99.133: icmp: echo request
>>>> 15:18:05.216021 f2:0b:a4:63:80:03 00:16:3e:be:ef:03 0800 98:
>>>> 172.16.99.133 > 172.16.99.203: icmp: echo reply
>>>>
>>>> That's on machine B.
>>>>
>>>> On machine A (the one replying to the ping), # tcpdump -peni tap100
>>>>
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>> listening on tap100, link-type EN10MB (Ethernet), capture size 65535 bytes
>>>>
>>>> machine A is a 5.99.55 kernel+world built  Tue Aug 23 12:05:48 UTC 2011,
>>>> tap100 is on bridge3 which is passed as an interface to machine B (a Xen
>>>> domU).
>>>
>>> What are the ifconfig outputs on the "machines" in question ?
>>
>> Machine B (OpenBSD):
>>
>> em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>        lladdr 00:16:3e:be:ef:03
>>        description: Management interface
>>        priority: 0
>>        media: Ethernet autoselect (1000baseT full-duplex)
>>        status: active
>>        inet 162.16.99.203 netmask 0xffffff00 broadcast xxx.xxx.xx.255
>>
>> Machine A (dom0; NetBSD current)
>>
>> tap100: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>>        address: f2:0b:a4:3c:ba:02
>>        media: Ethernet autoselect
>>        inet 172.16.99.133 netmask 0xffffff00 broadcast 172.16.99.255
>>
>>
>> bridge3: flags=41<UP,RUNNING>
>>        Configuration:
>>                priority 32768 hellotime 2 fwddelay 15 maxage 20
>>                ipfilter disabled flags 0x0
>>        Interfaces:
>>                tap3 flags=3<LEARNING,DISCOVER>
>>                        port 18 priority 128
>>                xvif1i3 flags=3<LEARNING,DISCOVER>
>>                        port 14 priority 128
>>                tap100 flags=3<LEARNING,DISCOVER>
>>                        port 5 priority 128
>>        Address cache (max cache: 100, timeout: 1200):
>>                00:16:3e:be:ef:03 tap3 385 flags=0<>
> 
> What is tap3 for, and why isn't there a real ethernet device in this
> bridge ?
>

tap3 is a xen-created interface.

>>>
>>>> Shouldn't the ping reply appear on tap100 @ the host ?
>>>
>>> Nope.
>>>
>>> You *really* don't want to send any packets that use the ethernet
>>> address of the tap device itself, they will not get bridged to
>>> anywhere that a socket is listening.
> 
>> On OpenBSD, I see @ the arp cache table:
>> ? (172.16.99.133) at f2:0b:a4:3c:ba:02 on em3
>>
>> And I'm logged in on machine A, through the IP assigned on tap100 there
>> (which is bridged via bridge3).
> 
> You don't want to assign an IP address to the tap device. I'm also
> not sure what you mean when you say that you are logged in through
> this address.
>

If not tap, then what should I use on NetBSD?

On OpenBSD I use
http://www.openbsd.org/cgi-bin/man.cgi?query=vether&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
(not in this case, but same general principle), and on FreeBSD I've
always used tap for related network voodoo.

> What real network interfaces are on the dom0 ?
> 

An internet connection, and a private switch connection. This
tap100/bridge3 interface can be conceived of as a private virtual switch.

>> It works as expected, it's just that I can't see any packets to tap100,
>> and tcpdump doesn't work on bridges (it seems).
> 
> I have some local diffs to bridge(4) to add support for bpf.
> 
> The tap(4) device is intended for use by machine emulators, it isn't
> a general debug interface.
> 
>> I may be missing something obvious, but if tcpdump on a bridge doesn't
>> work and tcpdump tap100 comes in clear, even though I have a TCP
>> connection established/active to the IP assigned on it, how should one
>> go on about troubleshooting connectivity (this is what brought me to ask
>> the question; in the mean time I've solved the problem, was missing a
>> static route, but tcpdump on tap100 still shows nothing) issues?
>>
>> In this particular case, the packets don't really come from any "real"
>> interface - thus the most I could do was, from machine B, confirming
>> that the packets were leaving the system. No way to actually confirm
>> they were coming in to machine A (hence tcpdump tap100)
> 
> The suggested setup for a dom0 is to bridge between the xvif device
> and a real ethernet one, you could then run tcpdump on the real
> interface.

Barring some more network voodoo, I would have to expose dom0 on the
internal network, which is what this setup avoids; The merits are
debatable, but to me it seems cleaner. Packets from my VPN address go
through the OpenBSD domU, then are injected on it's em3 interface, which
is on the same bridge as tap100 on the dom0. Since tap100 has an IP
address assigned, I can login to the dom0 using that.

> 
> You shouldn't need a static route.
> 

In this configuration I do; What I'm doing here is not having sshd
listen on the internet side or the private switch side on the dom0; I
have bridge3 on the dom0, where the dom0 attaches (via tap100, with an
IP assigned), but so does the domU. The domU then routes packets coming
in from my OpenVPN IP address to the dom0 tap100 IP address.

That is why the static route was required - the dom0 didn't know how to
reach the OpenVPN IP range. (note: the domU is also the VPN server)

>> Am I making sense?
> 
> You are, I think we may need to improve some documentation though.
> 
> Robert Swindells