Re: NetBSD/xen goes off the network - reproduceable

[Brian Marcotte <marcotte%panix.com@localhost>]

> > xennet_handler: copying packet: i=902 free_rxreql=0
> > xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
> > xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=1
> 
> but there is again one slot in the ring; this should be enough to make
> limited progress.

Another test with the same kernel looked like this when it locked up:

    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
    xennet_handler: copying packet: i=961 free_rxreql=0
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
    xennet_handler: copying packet: i=962 free_rxreql=0
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0

> > # pkill telnet
> And at this point, the network has not restarted ?

Sorry if that wasn't clear. When I killed telnet, the network came
back immediately.

> When this happen, can you check the flags of the correponding
> xvif interface in the backend ?

I'm not sure if I understand your request. If you mean output from
ifconfig, it looks like this:

    vif13.0   Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
              inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
              UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
              RX packets:459 errors:0 dropped:0 overruns:0 frame:0
              TX packets:890 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:32
              RX bytes:30807 (30.0 KiB)  TX bytes:79021 (77.1 KiB)

Other than the packet counters, it is always the same.

If you mean the output from netstat -i, it looks like this:

Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
vif9.0 1500 0       497      0      0 0           963      0      0      0 BMPRU

Normally, the packet counters are increasing. When the network stops, so
do the packet counters in the dom0.

> Can you give details about your dom0 ?

Currently I have Linux 2.6.18 on Xen 3.3 and Linux 3.1 on Xen 4.1. NetBSD
guests behave the same on both of these systems.

-----------------------

I have also repeated these tests with feature-rx-notify turned off.
In this case, things are a little better, but still unusable for heavy
network loads like NFS.

The complete console log is here:

        http://www.panix.com/~marcotte/consolelog-no-rx-notify.txt

The difference is that instead of the network stopping completely, the
dom0 start dropping packets. This can be seen in the dom0 by running
netstat -i repeatedly:

Iface    MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
vif11.0 1500 0       520      0      0 0         1182      0     21      0 BMPRU
vif11.0 1500 0       528      0      0 0         1195      0     26      0 BMPRU
vif11.0 1500 0       542      0      0 0         1215      0     28      0 BMPRU
vif11.0 1500 0       557      0      0 0         1240      0     52      0 BMPRU

On the console:

    xennet_rx_mbuf_free: sc->sc_free_rxreql=5 SC_NLIVEREQ(sc)=7
    xennet_rx_mbuf_free: sc->sc_free_rxreql=6 SC_NLIVEREQ(sc)=4
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=7
    xennet_rx_mbuf_free: sc->sc_free_rxreql=1 SC_NLIVEREQ(sc)=1
    xennet_handler: copying packet: i=1074 free_rxreql=0
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
    xennet_handler: copying packet: i=1075 free_rxreql=0
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
    xennet_handler: copying packet: i=1076 free_rxreql=0

    [ It stays with this pattern until the test is stopped ]

    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=0
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=1
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=2
    xennet_rx_mbuf_free: sc->sc_free_rxreql=1 SC_NLIVEREQ(sc)=2
    xennet_rx_mbuf_free: sc->sc_free_rxreql=0 SC_NLIVEREQ(sc)=4
    ...
    xennet_rx_mbuf_free: sc->sc_free_rxreql=93 SC_NLIVEREQ(sc)=162
    xennet_rx_mbuf_free: sc->sc_free_rxreql=94 SC_NLIVEREQ(sc)=161
    xennet_rx_mbuf_free: sc->sc_free_rxreql=95 SC_NLIVEREQ(sc)=160

Thanks

--
- Brian