Re: xen and networking problems

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Fri, Oct 30, 2009 at 02:43:26PM -0600, Brook Milligan wrote:
> Manuel Bouyer writes:
>  > Please check with netstat if you have interrupt for this device.
> 
> # vmstat -i
> interrupt                                     total     rate
> vcpu0 ioapic0 pin 17                           1205        9
> vcpu0 ioapic0                                  2644       21
> vcpu0 ioapic0                                    10        0
> vcpu0 ioapic0                                   485        3
> vcpu0 clock                                   13057      104
> vcpu0 xenbus                                     53        0
> Total                                         17454      139
> 
> Note that the ioapic0 pin 17 device corresponds to the fxp0 network
> interface.  This command was run while pinging the interface, and the
> number of interrupts increases each time the command is run.  I
> presume this means that the hardware is generating interrupts as
> packets arrive.

So interrupts are probably working

> 
> In addition, the results of netstat are below.  The number of packets
> in increased with each run of the command and the number of packets
> out increased if I pinged from this machine (although outgoing pings
> always resulted in 'ping: sendto: Host is down' errors).
> 
> # netstat -i
> Name  Mtu   Network       Address              Ipkts Ierrs    Opkts Oerrs 
> Colls
> fxp0  1500  <Link>        00:30:48:23:3a:81     1702     0       14     0     > 0
> fxp0  1500  176.16.0.2/24 172.16.0.2            1702     0       14     0     > 0
> fxp0  1500  fe80::/64     fe80::230:48ff:fe     1702     0       14     0     > 0
> lo0   33184 <Link>                                 0     0        0     0     > 0
> lo0   33184 127/8         localhost                0     0        0     0     > 0
> lo0   33184 localhost/128 ::1                      0     0        0     0     > 0
> lo0   33184 fe80::/64     fe80::1                  0     0        0     0     > 0
> 
>  > You could also run tcpdump and see if you receive some packets.
> 
> tcpdump never captured any packets under these tests, which were all
> done without creating the bridge0 interface.  (The bridge was present
> when I generated output for earlier posts.)  Before I disabled
> bridge0, however, tcpdump reported only STP packets.  For these tests
> I disabled the bridge to make sure that it wasn't interfering with the
> primary network interface.

Did you run it with -n ? Otherwise, IP reverse resolution could cause it
to stall.

> 
> Does all of this mean that the packets are arriving at the hardware
> interface, are being partially processed by the kernel network stack,
> but are being lost somewhere along the way before making it to any
> applications?

tcpdump being silent makes it worse than that, as the packets are
passed to bpf from the driver itself, in the interrupt routine.


> 
> What else can be done to track down what is going on here?

check that it's not running out of mbufs. vmstat -m will tell you
if mbuf allocation fail (mbuf and mbuf cluster). 

> 
> By the way, without a serial console is there a way to capture the
> boot messages from the xen hypervisor?  Alternatively, is there a way
> to pause the boot process so I can review the messages before they
> scroll off the screen?

'xm dmesg' should give them to you

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--