Re: xen interrupt system

[Christoph Egger <Christoph_Egger%gmx.de@localhost>]

On Friday 21 August 2009 11:32:14 Manuel Bouyer wrote:
> On Fri, Aug 21, 2009 at 08:17:27AM +0200, Christoph Egger wrote:
> > > > (XEN) ioapic_guest_write: Installing bogus unmasked IO-APIC entry!
> >
> > There's a C comment right before this message:
> >
> > /* This pin leads nowhere but the guest has not masked it. */
> >
> > When we this message, then Xen didn't really allocate the interrupt for
> > us since it does a more strict validation check if the irq we want to
> > allocate is really an ioapic interrupt. irq 94 is not an existing ioapic
> > interrupt on that machine.
>
> when interrupts are routed using apics, IRQs are not used at the
> hardware level (it's the vector programmed in the apic registers that
> is important), and the hardware doesn't even know the IRQ number we use.
> This is a software object, used to index the interrupt vector arrays
> between Xen and the dom0.
>
> > > > ioapic0: int7
> > > > 0x1a95e<vector=0x5e,delmode=0x1,logical,actlo,level,masked,dest=0x0>
                                                ^^^^
> > > > 0xff000000<target=0xff>
> > > > svwsata0: using ioapic0 pin 7, event channel 7 for native-PCI
> > > > interrupt

XenSource:

"Well, dom0 may think it using a vector but it isn't. Linux-2.6.18 for
example is allocating vectors for irqs using PHYSDEVOP_alloc_irq_vector(),
as it is supposed to. But crucially after Xiantao's patch it now gets the
irq returned back to it. Dom0 then thinks it is poking a vector into the
IO-APIC RTE, but Xen knows it is an irq and does the appropriate
translation. We can do tricks like this because Xen manages the vector
'namespace' and so can virtualise it as it likes.

So I suppose the question is -- where is NetBSD dom0 conjuring vector
numbers from? You're going to have to track down the source of that 'vector'
0x5e in the NetBSD kernel I think."

XenSource thinks that the vector 0x5e is wrong because according
to this xen boot messages, the vectors go from 0 - 47.

(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
(XEN) IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
(XEN) ACPI: IOAPIC (id[0x0a] address[0xfec02000] gsi_base[32])
(XEN) IOAPIC[2]: apic_id 10, version 17, address 0xfec02000, GSI 32-47
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)

> > > We had an active entry for this pin. I guess Xen wants us to mask it
> > > before changing it (but a look in the sources would tell more), this is
> > > what the code in ioapic_attach() tries to do (in fact, it reprograms
> > > the apic from scratch).
> > > The old entry was IOAPIC_REDLO_DSTMOD, IOAPIC_REDLO_DEL_LOPRI and we're
> > > trying to add the IOAPIC_REDLO_ACTLO | IOAPIC_REDLO_LEVEL flags (in
> > > addition to changing the vector). I don't know why Xen says it's
> > > invalid, a look at the sources could give details about this.
> >
> > Xen's recent interrupt change happened to support per-cpu vectors. I
> > suppose we need to change our way in allocating interrupts.
>
> that's possible. But we have to be carefull to not break older hypervisors.
> With Xen 3.0 I couldn't recycle low interrupt numbers, because Xen uses
> some of them and I don't know which.