interrupt misrouting and storms in dom0?

To: port-xen%netbsd.org@localhost
Subject: interrupt misrouting and storms in dom0?
From: Thor Lancelot Simon <tls%rek.tjls.com@localhost>
Date: Tue, 28 Apr 2009 11:23:18 -0400
I have a system which seems to generate a huge number of spurious
interrupts in the dom0 under load.  I should note that this system
requires ACPI to be disabled in order to run properly *without* xen
but I have never figured out why (mpt seems to never get interrupts).
With xen, device interrupts are delivered -- oh, are they ever!

With xen, I see symptoms like this under disk activity:

interrupt                                     total     rate
vcpu0 ioapic0 pin 17                       29337995     4079
vcpu0 ioapic0 pin 18                      111999578    15572
vcpu0 ioapic0 pin 23                             15        0
vcpu0 ioapic0                                     9        0
vcpu0 clock                                 1426216      198
vcpu0 xenbus                                   1363        0
Total                                     142765176    19850

If I do a find or rm -r I can get pin 18 to spike as high as 70,000/sec.
The disk is never clearing more than about 100tps.

What is more confusing is that:

1) ioapic0 pin 18 isn't even connected to the disk controller according to
   our boot-time diagnostics.  It's connected to uhci, which is not in use.

2) ioapic0 pin 17 *is* connected to the disk controller.  When the system is
   totally idle, it reports 2000-4000 interrupts/sec all the time.

Any idea what's going on?  The kernel is:
         NetBSD 5.99.11 (XEN3_DOM0) #0: Thu Apr 23 04:08:46 UTC 2009
from the autobuild cluster.  Dmesg snippets follow:

 Xen version 3.1.4 (@) (gcc version 4.1.3 20080704 prerelease (NetBSD nb2 
20081120)) Thu Apr 23 01:19:11 UTC 2009
 Latest ChangeSet: unavailable
[...]
(XEN) Processor #0 15:4 APIC version 20
(XEN) Processor #6 15:4 APIC version 20
(XEN) Processor #1 15:4 APIC version 20
(XEN) Processor #7 15:4 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 32-55
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec83000, GSI 64-87
(XEN) Enabling APIC mode:  Flat.  Using 3 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3192.316 MHz processor.
(XEN) CPU0: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 1/6 eip 90000
(XEN) CPU1: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 2/1 eip 90000
(XEN) CPU2: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 3/7 eip 90000
(XEN) CPU3: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
[...]
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=0, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
Dmesg snippets follow:

(XEN) Processor #0 15:4 APIC version 20
(XEN) Processor #6 15:4 APIC version 20
(XEN) Processor #1 15:4 APIC version 20
(XEN) Processor #7 15:4 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 32-55
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec83000, GSI 64-87
(XEN) Enabling APIC mode:  Flat.  Using 3 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3192.316 MHz processor.
(XEN) CPU0: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 1/6 eip 90000
(XEN) CPU1: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 2/1 eip 90000
(XEN) CPU2: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Booting processor 3/7 eip 90000
(XEN) CPU3: Intel(R) Xeon(TM) CPU 3.20GHz stepping 0a
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
[...]
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=0, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
[...]
ioapic0 at mainbus0 apid 8: pa 0xfec00000, virtual wire mode, version 20, 24
pins
ioapic0: can't remap to apid 8
ioapic1 at mainbus0 apid 9: pa 0xfec80000, virtual wire mode, version 20, 24
pins
ioapic1: can't remap to apid 9
ioapic2 at mainbus0 apid 10: pa 0xfec83000, virtual wire mode, version 20, 24
pins
ioapic2: can't remap to apid 10
ipmi0 at mainbus0
hypervisor0 at mainbus0: Xen version 3.1
vcpu0 at hypervisor0: Intel 686-class, 3192MHz, id 0xf4a
debug virtual interrupt using event channel 1
xenbus0 at hypervisor0: Xen Virtual Bus Interface
xencons0 at hypervisor0: Xen Virtual Console Driver
acpi0 at hypervisor0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <DELL  ,PE BKC  ,00000001>, AslId <MSFT,0100000a>
mpacpi: found root PCI bus 0 at level 1
mpacpi: found subordinate bus 1 at level 2
mpacpi: failed to derive bus number, ignoring
mpacpi: failed to derive bus number, ignoring
mpacpi: found subordinate bus 4 at level 2
mpacpi: failed to derive bus number, ignoring
mpacpi: found subordinate bus 5 at level 2
mpacpi: found subordinate bus 6 at level 3
mpacpi: found subordinate bus 7 at level 3
mpacpi: failed to derive bus number, ignoring
mpacpi: failed to derive bus number, ignoring
mpacpi: failed to derive bus number, ignoring
mpacpi: found subordinate bus 9 at level 2
mpacpi: 7 PCI busses
mpacpi: ISA interrupt override 0 -> 2 (0/0)
mpacpi: ISA interrupt override 9 -> 9 (1/3)
mpacpi: configuring PCI bus 0 int routing
mpacpi: configuring PCI bus 1 int routing
mpacpi: configuring PCI bus 4 int routing
mpacpi: configuring PCI bus 5 int routing
mpacpi: configuring PCI bus 6 int routing
mpacpi: configuring PCI bus 7 int routing
mpacpi: configuring PCI bus 9 int routing
ioapic0: pin 2 attached to isa0 irq 0 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 1 attached to isa0 irq 1 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 3 attached to isa0 irq 3 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 4 attached to isa0 irq 4 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 5 attached to isa0 irq 5 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 6 attached to isa0 irq 6 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 7 attached to isa0 irq 7 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 8 attached to isa0 irq 8 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 9 attached to isa0 irq 9 (type 0x0<type=0x0> flags
0xd<pol=0x1=Act Hi,trig=0x3=Level>)
ioapic0: pin 10 attached to isa0 irq 10 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 11 attached to isa0 irq 11 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 12 attached to isa0 irq 12 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 13 attached to isa0 irq 13 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 14 attached to isa0 irq 14 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 15 attached to isa0 irq 15 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 16 attached to pci0 device 2 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 3 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 4 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 5 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 6 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 7 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 29 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci0 device 29 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci0 device 29 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 23 attached to pci0 device 29 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci1 device 0 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci1 device 0 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci1 device 0 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci1 device 0 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci4 device 0 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci4 device 0 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci4 device 0 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci4 device 0 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci5 device 0 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci5 device 0 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci5 device 0 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci5 device 0 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic2: pin 0 attached to pci6 device 7 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic2: pin 1 attached to pci7 device 8 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 20 attached to pci9 device 5 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 21 attached to pci9 device 5 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 23 attached to pci9 device 6 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci9 device 13 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci9 device 3 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci9 device 3 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci9 device 3 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci9 device 3 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci9 device 1 INT_A (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci9 device 1 INT_B (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci9 device 1 INT_C (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci9 device 1 INT_D (type 0x0<type=0x0> flags
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: int9 0x189a0<vector=0xa0,delmode=0x1,logical,level,masked,dest=0x0>
0xf000000<target=0xf>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
ACPI-Safe 24-bit timer
[...]
pci2: i/o space, memory space enabled
mpt0 at pci2 dev 5 function 0: vendor 0x1000 product 0x0030
ioapic0: int17 
0x1a9a8<vector=0xa8,delmode=0x1,logical,actlo,level,masked,dest=0x0> 
0xf000000<target=0xf>
mpt0: interrupting at ioapic0 pin 17, event channel 5
scsibus0 at mpt0: 16 targets, 8 luns per target
[...]
ioapic0: int18 
0x1a9d0<vector=0xd0,delmode=0x1,logical,actlo,level,masked,dest=0x0> 
0xf000000<target=0xf>
uhci2: interrupting at ioapic0 pin 18, event channel 8

-- 
Thor Lancelot Simon                                        
tls%rek.tjls.com@localhost
    "Even experienced UNIX users occasionally enter rm *.* at the UNIX
     prompt only to realize too late that they have removed the wrong
     segment of the directory structure." - Microsoft WSS whitepaper
Follow-Ups:
- Re: interrupt misrouting and storms in dom0?
  - From: Thor Lancelot Simon
Prev by Date: Re: PCI passthrough not working?
Next by Date: telling real from HTT CPUs under xen?
Previous by Thread: Some questions about x86/xen pmap
Next by Thread: Re: interrupt misrouting and storms in dom0?
Indexes:
Home | Main Index | Thread Index | Old Index