Re: bnx driver in 5.99.7 panics under xen 3.3.1

[Mike Bowie <mbowie%buzmo.com@localhost>]

Manuel Bouyer wrote:
> On Mon, Feb 02, 2009 at 12:16:04PM -0800, Mike Bowie wrote:
>>> it looks like an interrupt storm. you may want to try playing with the
>>> acpi options ...
>>>
>> Sadly, my hacking over the weekend has not proven all that successful.
>> Adding "acpi=off" and a dom0 kernel without ACPI has allowed dom0 to
>> boot, but not reliably.  It does appear to be an interrupt storm, but I
>> don't know what more I can do to calm it.
>>
>> When the machine *does* boot, I'm unable to login on the serial console,
>> as just entering a username will grind the machine to a halt.  If I
>> login via ssh, top shows interrupts constantly between 7% and 15% and
>> the machine's responsiveness is sluggish to say the least.
>
> What does 'systat vm' show ? This would probably show you which interrupt
> is generating the load.
> You can also try entering the debugger with +++++
> and type 'show events' from the db> prompt.
>

"ioapic0 pin 11" is catching about 114000 interrupts every two or three
seconds.

# dmesg | grep ioapic0
ioapic0 at mainbus0 apid 8: pa 0xfec00000, virtual wire mode, version
11, 16 pins
ioapic0: can't remap to apid 8
ioapic0: int0 attached to ExtINT (type 0x3<type=0x3=ExtINT> flags
0x5<pol=0x1=Act Hi,trig=0x1=Edge>)
ioapic0: int1 attached to isa0 irq 1 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int2 attached to isa0 irq 0 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int3 attached to isa0 irq 3 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int4 attached to isa0 irq 4 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int6 attached to isa0 irq 6 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int7 attached to isa0 irq 7 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int8 attached to isa0 irq 8 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int9 attached to isa0 irq 9 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int12 attached to isa0 irq 12 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int14 attached to isa0 irq 14 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int15 attached to isa0 irq 15 (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int11 attached to pci0 device 3 INT_A (type 0x0<type=0x0> flags
0x0<pol=0x0,trig=0x0>)
ioapic0: int11 attached to pci9 device 14 INT_A (type 0x0<type=0x0>
flags 0x0<pol=0x0,trig=0x0>)
ioapic0: int11
0x1a9b0<vector=0xb0,delmode=0x1,logical,actlo,level,masked,dest=0x0>
0xff000000<target=0xff>
svwsata0: using ioapic0 pin 11, event channel 3 for native-PCI interrupt
ohci0: interrupting at ioapic0 pin 11, event channel 3
ohci1: interrupting at ioapic0 pin 11, event channel 3
ehci0: interrupting at ioapic0 pin 11, event channel 3

db> show event
evcnt type 0: bus_dma loads = 5449
evcnt type 0: bus_dma nbouncebufs = 2062
evcnt type 0: vmcmd kills = 175
evcnt type 0: vmcmd extends = 11
evcnt type 0: vmcmd calls = 1498
evcnt type 0: softint net/0 = 629
evcnt type 0: softint bio/0 = 5311
evcnt type 0: softint clk/0 = 16535
evcnt type 0: softint ser/0 = 2
evcnt type 0: callout late/0 = 18
evcnt type 0: crosscall unicast = 2
evcnt type 0: namecache entries collected = 282
evcnt type 0: namecache under scan target = 997
evcnt type 0: nfs timer = 8
evcnt type 0: nfs timer start = 2
evcnt type 0: nfs timer stop = 2
evcnt type 1: vcpu0 xencons = 5
evcnt type 1: vcpu0 ioapic0 pin 11 = 110038846
evcnt type 1: vcpu0 ioapic1 pin 5 = 6275
evcnt type 1: vcpu0 clock = 484635
evcnt type 1: vcpu0 xenbus = 1

>> Xen's inability to reboot this box makes it a bit of a dog to
>> troubleshoot too; it would probably be easier if it was local rather
>> than at the co-lo.  Each crash requires that I reset the remote DRAC
>> card (because the interrupt storms seem to disable it's com port) wait
>> for it to come back up and issue a hard reset before returning to a
>> plain NetBSD kernel.
>
> Would
> ^A^A^AR (or ^A^A^Ar)
> work to reboot from xen ? ^A^A^A switches console input to Xen, and
> R (or r, I never remember) ask xen to reboot hard.
>

Xen itself is unable to reboot the domain in such as fashion, as was
NetBSD prior to the patch Christos committed to use the reset control
register.  I did look at adding the key statements to
/xen/arch/x86/shutdown.c but my copy and paste resulted in the compiler
complaining that the integer was being truncated, so I moved on.

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/amd64/amd64/machdep.c.diff?r1=1.119&r2=1.120&only_with_tag=MAIN&f=h

Cheers,

Mike.