MSI/MSI-X implementation and interrupt handling on i386/amd64

[Geoff Wing <gcw%pobox.com@localhost>]

Hi,
brief background:  on an amd64 VM (1 CPU on VMWare ESXi) I had a network
interface (vmx) failing because it could not get an interrupt slot.  The
vmx wants 3 interrupts per interface (tx/rx/link-state).  I had a few
on an admin machine and one started failing when ahcisata was changed to
use MSI (not ahcisata's fault, obviously).

On i386/amd64 each CPU has a 32 bitmask for interrupts (1 bit per) - but
16 of the 32 are reserved for legacy IRQs (on the first CPU).  MSI-X allows
for 2048 interrupts.  On a physical machine with many CPUs the MSI interrupts
are farmed out across the different CPUs so would not be apparent to most.
(and no problem for those 65+ core machines).

For my personal use, I've hacked around by ignoring the reserved legacy IRQ
region because it's not relevant to me in my VM with MSI/MSI-X.  Other
people using single CPU VMs may start bumping into this issue so just
making people aware.  I haven't looked into changing how interrupts are
handled or if there would be significant performance penalty.

Regards,
Geoff

FYI (pin17 is mpt0):
% intrctl list
interrupt id      CPU0  device name(s)
ioapic0 pin 9        0* acpi SCI
ioapic0 pin 1        0* pckbc1 kbd
ioapic0 pin 12       0* pckbc2 aux
ioapic0 pin 14       0* piixide0 primary
ioapic0 pin 15       0* piixide0 secondary
ioapic0 pin 17 3481843* unknown
ioapic0 pin 18      54* uhci0
ioapic0 pin 19       0* ehci0
msi0 vec 0           0* ahcisata0
msix1 vec 0      16215* vmx0: tx 0
msix1 vec 1     406335* vmx0: rx 0
msix1 vec 2          0* vmx0: link
msix2 vec 0     100571* vmx1: tx 0
msix2 vec 1     178436* vmx1: rx 0
msix2 vec 2          0* vmx1: link
msix3 vec 0     327583* vmx2: tx 0
msix3 vec 1    3141480* vmx2: rx 0
msix3 vec 2          0* vmx2: link