Re: xen interrupt system

[Christoph Egger <Christoph_Egger%gmx.de@localhost>]

On Friday 28 August 2009 15:28:12 Manuel Bouyer wrote:
> On Fri, Aug 28, 2009 at 02:55:02PM +0200, Christoph Egger wrote:
> > On Tuesday 25 August 2009 20:39:32 Manuel Bouyer wrote:
> > > On Tue, Aug 25, 2009 at 02:34:11PM +0200, Christoph Egger wrote:
> > > > > > > How do we know the "GSI space" ?
> > > > > >
> > > > > > From the MADT ACPI table, IO Apic structure byte offset 8.
> > > > >
> > > > > And if we don't use ACPI ? I have systems where it's not useable.
> > > >
> > > > a) Update your BIOS to latest version.
> > >
> > > I have the latest version.
> > >
> > > > b) Not sure how many such systems would have more than one IOAPIC. If
> > > > there's only one IOAPIC then it's easy as its gsi_base will be zero.
> > >
> > > The Dell PE1850 have several ioapics for example.
> > >
> > > > > I guess we have something similar in the MPBIOS ?
> > > > > I think we have the GSI in pic_vecbase in NetBSD, but I suspect
> > > > > it's set to -1 when using MPBIOS (from reading the sources).
> > > >
> > > > The issue of no ACPI is hardly a practical one these days, as there
> > > > aren't really any system made in the last 10+ years with no ACPI.
> > >
> > > The PE1850 isn't 10 years old, and yet it has its issues with acpi.
> > >
> > > > > Now back to the original question, what does
> > > > > PHYSDEVOP_ASSIGN_VECTOR expects as input ?
> > > >
> > > > According to Intel, it always expected the GSI number.
> > >
> > > Experimentally, it was accepting any number, unless it was a
> > > number already allocated (and some were allocated at boot).
> > > Maybe we can change it to try to use the GSI number when it exists,
> > > hopefully we'll reuse the same number as Xen already used for this pin
> > > and things would work for older hypervisors too.
> >
> > Intel gave me a patch to test. It makes PHYSDEVOP_allocate_irq_vector
> > a noop and makes Xen to allocate the irq "on demand" when writing
> > an ioapic rte. This makes the hypercall succeed in general and NetBSD
> > wants to bind the irq number 200 to an event channel. The
> > EVTCHN_bind_pirq fails with EINVAL. This in turn leads to this panic
> > below.
>
> Yes, of course.
>
> > XenSource points out, EVTCHN_bind_pirq expects pirq number
> > within the GSI space. On the test machine, this GSI space goes from 0 to
> > 47.
>
> This is what changed. With previous version EVTCHN_bind_pirq woud accept
> any IRQ number, as long as it was unused. We need to rework this part.
> The question is 1) if we can guess the corret GSI in all cases (I'm not
> sure) 2) if this will work with pre-3.4 hypervisors.

Intel sent me a new patch to test which maps the value passed 
PHYSDEVOP_alloc_irq_vector with EVTCHN_bind_pirq internally, so that the
value passed to EVTCHN_bind_pirq is "recognized" by Xen when it allocates
the real interrupt "on demand" on IOAPIC writes.

Now this makes the NetBSD dom0 fail to boot in a different way:
Our way to initialize ioapics fails.

ioapic0 at mainbus0 apid 8, virtual wire mode
(XEN) io_apic.c:2196: 
(XEN) ioapic_guest_write: apic=0, pin=0, irq=0
(XEN) ioapic_guest_write: new_entry=00010900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2196: 
(XEN) ioapic_guest_write: apic=0, pin=2, irq=0
(XEN) ioapic_guest_write: new_entry=00010900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) irq.c:1297: dom0: pirq 0 or irq 3 already mapped
PHYSDEVOP_APIC_WRITE ret -22
panic: PHYSDEVOP_APIC_WRITE
fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip ffffffff8020aa8d cs 10000e030 rflags 46 cr2  0 cpl 8 
rsp ffffffff80f00ab0
Stopped in pid 0.1 (system) at  netbsd:breakpoint+0x5:  leave
breakpoint() at netbsd:breakpoint+0x5
panic() at netbsd:panic+0x285
ioapic_attach() at netbsd:ioapic_attach+0x621
config_attach_loc() at netbsd:config_attach_loc+0x173
mpacpi_config_ioapic() at netbsd:mpacpi_config_ioapic+0x5e
acpi_madt_walk() at netbsd:acpi_madt_walk+0x45
mpacpi_scan_apics() at netbsd:mpacpi_scan_apics+0xa0
mainbus_attach() at netbsd:mainbus_attach+0x129
config_attach_loc() at netbsd:config_attach_loc+0x173
cpu_configure() at netbsd:cpu_configure+0x25
main() at netbsd:main+0x1aa


Intel and XenSource also stated:

Intel:
"For MSI, we need to solve the pirq conflict issue, since the irq passed by
PHYSDEVOP_alloc_irq_vector is recognized as pirq number in the
EVTCHN_bind_pirq."

XenSource (in response to Intel's statement):
"My guess is that NetBSD does not have support for MSI on Xen so that's a
non-issue. And Linux provides GSI to alloc_irq_vector and to bind_pirq (when
setting up legacy INTx irqs) so obviously avoids the possible conflict. If
NetBSD supports MSI-on-Xen in future, it will have to clean up its act
regarding the irq numbers it hands to the abovementioned hypercalls. That's
a reasonable constraint though."

Christoph