Re: PCI passthru with Xen 3.0, NetBSD 5.1 dom0, Linux domU

[Konrad Rzeszutek Wilk <konrad.wilk%oracle.com@localhost>]

On Tue, Jun 07, 2011 at 07:27:01PM +0200, Manuel Bouyer wrote:
> On Tue, Jun 07, 2011 at 06:21:43PM +0200, Julien Oster wrote:
> > Hi Konrad,
> > 
> > On Mon, Jun 6, 2011 at 17:59, Konrad Rzeszutek Wilk
> > <konrad.wilk%oracle.com@localhost> wrote:
> > 
> > > Are the IRQs being delievered to DomU? You can check the /proc/interrupts.
> > 
> > Apparently, yes. They appear in /proc/interrupts and the card seems to
> > indeed do its job.
> > 
> > > Is it using MSI? I am not sure if NetBSD pciback supports MSI and if that
> > > is what it tries it might have choked on something (and I remember
> > > writting some patches for xen-pcifront to fix misbehaving backends).
> > > I think they went in 2.6.39
> > 
> > Hmm, good question. The card is certainly new enough. pcictl dump said:
> > 
> >    Interrupt pin: 0x01 (pin A)
> >    Interrupt line: 0xc2
> > 
> > ... but that doesn't mean the device couldn't trigger MSI interrupts
> > anyway, maybe additionally, I guess? The next time I boot the domU
> > with PCI passthru, I'll take a look at exact kernel messages and poke
> > around a bit to see if I can find out what it uses.
> > 
> > In any way, I played around a bit and so far everything really seemed
> > to work well. At least I was able to associate to an AP at home (and
> > authenticate via WPA), and also make it act as one or even several APs
> > itself, using hostapd. Apart from at scan time, it rarely produced the
> > "Failed to stop TX DMA" messages and so far, I could not see any ill
> > effects. Wasn't a thorough test, though.
> > 
> > What did go wrong, however, was that "sometimes" (don't know what the
> > exact circumstances are, yet), dom0 would hang in a very peculiar
> > manner just after, or when completing a shutdown of the Linux domU.
> > Userspace seemed gone in the manner that any ssh session wasn't
> > responding in the slightest anymore, but kernel was still somehow
> > alive, giving replies to ICMP echoes after several *seconds*. I
> > currently have neither a VGA nor a serial console connected to the
> > machine, so I had to power cycle each time.
> 
> Is the interrupt used by this card shared with other devices driven by
> the dom0 ? If so, I guess this could explain it: the adapter generates
> an interrupt, and as the domU is not running anymore nothing is there to

Hmm, most of the time the guest is pretty good at actually disabling
the device. Could it be that the PCI backend in NetBSD not doing the right
cleanup?
> ack it. As the interrupt is shared with the dom0, the dom0 gets an
> interrupt storm.

Ugh, that woudl be indeed horrible. The Xen hypervisor should be still
running I think,so it should be possible to hit Ctrl-A couple of times
and find out what if the GSI for the PCI device is masked or not (and pending).
> 
> -- 
> Manuel Bouyer <bouyer%antioche.eu.org@localhost>
>      NetBSD: 26 ans d'experience feront toujours la difference
> --