>Number:         31359
>Category:       port-amd64
>Synopsis:       SMP amd64 system exhibits excessive cpu shootdown IPIs
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Sep 20 16:19:00 +0000 2005
>Originator:     oster@cs.usask.ca
>Release:        NetBSD 3.99.9
>Organization:
>Environment:
NetBSD quadrapus 3.99.9 NetBSD 3.99.9 (QUADRAPUS.ddb) #0: Mon Sep 19 13:58:19 CST 2005  root@quadrapus:/usr/local/current/src/sys/arch/amd64/compile/QUADRAPUS.ddb amd64
Architecture: amd64
Machine: amd64
>Description:

On a quad Opteron box under simple CPU load, the number of shootdown IPIs
seems to be quite high at 80000-250000 per second when doing something
as simple as building a GENERIC kernel.

select output from 'vmstat -i 1' while just building a GENERIC 
kernel with 'make -j 4':

interrupt                                     total     rate
cpu3 TLB shootdown IPI                    212144185     3020
cpu3 TLB shootdown IPI                    212288403     3022
cpu3 TLB shootdown IPI                    212375532     3023
cpu3 TLB shootdown IPI                    212492803     3025

Here, in the space of 4 seconds, 250000 IPIs have been seen.
The output from top during this is something like:

52 processes:  1 runnable, 47 sleeping, 4 on processor
CPU0 states: 20.9% user,  0.0% nice, 54.7% system, 21.9% interrupt,  2.5% idle
CPU1 states: 31.8% user,  0.0% nice, 67.2% system,  0.0% interrupt,  1.0% idle
CPU2 states: 31.5% user,  0.0% nice, 68.0% system,  0.0% interrupt,  0.5% idle
CPU3 states: 33.0% user,  0.0% nice, 64.0% system,  0.0% interrupt,  3.0% idle
Memory: 182M Act, 8504K Wired, 10M Exec, 125M File, 6573M Free

where much less "real work" is being done by the CPUs than expected.

As an indicationg that "something is wrong": a single Athlon XP 3000+ 
(2.2GHz) can cross-build an amd64 distribution *faster* than this 
quad Opteron 850 @ 2.4GHz can with a parallel build!

dmesg follows:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 3.99.9 (QUADRAPUS.ddb) #0: Mon Sep 19 13:58:19 CST 2005
        root@quadrapus:/usr/local/current/src/sys/arch/amd64/compile/QUADRAPUS.ddb
total memory = 7743 MB
avail memory = 7462 MB
mainbus0 (root)
mainbus0: Intel MP Specification (Version 1.4) (SUN      SunFire V40z)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Opteron(tm) Processor 850, 2393.33 MHz
cpu0: features: e7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features: e7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu0: features: e7dbfbff<FXSR,SSE,SSE2,LONG,3DNOW2,3DNOW>
cpu0: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu0: L2 cache 1 MB 64B/line 16-way
cpu0: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: calibrating local timer
cpu0: apic clock running at 199 MHz
cpu0: 16 page colors
cpu1 at mainbus0: apid 1 (application processor)
cpu1: starting
cpu1: AMD Opteron(tm) Processor 850, 2393.18 MHz
cpu1: features: e7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features: e7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu1: features: e7dbfbff<FXSR,SSE,SSE2,LONG,3DNOW2,3DNOW>
cpu1: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu1: L2 cache 1 MB 64B/line 16-way
cpu1: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu1: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu2 at mainbus0: apid 2 (application processor)
cpu2: starting
cpu2: AMD Opteron(tm) Processor 850, 2393.18 MHz
cpu2: features: e7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu2: features: e7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu2: features: e7dbfbff<FXSR,SSE,SSE2,LONG,3DNOW2,3DNOW>
cpu2: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu2: L2 cache 1 MB 64B/line 16-way
cpu2: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu2: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu3 at mainbus0: apid 3 (application processor)
cpu3: starting
cpu3: AMD Opteron(tm) Processor 850, 2393.18 MHz
cpu3: features: e7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu3: features: e7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu3: features: e7dbfbff<FXSR,SSE,SSE2,LONG,3DNOW2,3DNOW>
cpu3: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu3: L2 cache 1 MB 64B/line 16-way
cpu3: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu3: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
mpbios: bus 0 is type PCI   
mpbios: bus 1 is type PCI   
mpbios: bus 2 is type PCI   
mpbios: bus 3 is type PCI   
mpbios: bus 32 is type PCI   
mpbios: bus 33 is type PCI   
mpbios: bus 37 is type PCI   
mpbios: bus 41 is type PCI   
mpbios: bus 45 is type PCI   
mpbios: bus 49 is type ISA   
ioapic0 at mainbus0 apid 4 (I/O APIC)
ioapic0: pa 0xfec00000, version 11, 24 pins
ioapic1 at mainbus0 apid 5 (I/O APIC)
ioapic1: pa 0xe4000000, version 11, 4 pins
ioapic2 at mainbus0 apid 6 (I/O APIC)
ioapic2: pa 0xe4001000, version 11, 4 pins
ioapic3 at mainbus0 apid 7 (I/O APIC)
ioapic3: pa 0xe5b01000, version 11, 4 pins
ioapic4 at mainbus0 apid 8 (I/O APIC)
ioapic4: pa 0xe5b03000, version 11, 4 pins
ioapic5 at mainbus0 apid 9 (I/O APIC)
ioapic5: pa 0xe5b05000, version 11, 4 pins
ioapic6 at mainbus0 apid 10 (I/O APIC)
ioapic6: pa 0xe5b07000, version 11, 4 pins
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
ppb0 at pci0 dev 6 function 0: Advanced Micro Devices AMD8111 I/O Hub (rev. 0x07)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
ohci0 at pci1 dev 0 function 0: Advanced Micro Devices AMD8111 USB Host Controller (rev. 0x0b)
ohci0: interrupting at ioapic0 pin 19 (irq 11)
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: Advanced Micro OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci1 dev 0 function 1: Advanced Micro Devices AMD8111 USB Host Controller (rev. 0x0b)
ohci1: interrupting at ioapic0 pin 19 (irq 11)
ohci1: OHCI version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: Advanced Micro OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
vga0 at pci1 dev 5 function 0: Trident Microsystems product 0x9880 (rev. 0x3a)
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
pcib0 at pci0 dev 7 function 0
pcib0: Advanced Micro Devices AMD8111 LPC Controller (rev. 0x05)
viaide0 at pci0 dev 7 function 1
viaide0: Advanced Micro Devices AMD8111 IDE Controller (rev. 0x03)
viaide0: bus-master DMA support present
viaide0: primary channel configured to compatibility mode
viaide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at viaide0 channel 0
viaide0: secondary channel configured to compatibility mode
viaide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at viaide0 channel 1
Advanced Micro Devices AMD8111 ACPI Controller (miscellaneous bridge, revision 0x05) at pci0 dev 7 function 3 not configured
ppb1 at pci0 dev 10 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
bge0 at pci2 dev 2 function 0: Broadcom BCM5703X Gigabit Ethernet
bge0: interrupting at ioapic1 pin 1 (irq 5)
bge0: ASIC BCM5703 A2 (0x1002), Ethernet address 00:09:3d:12:ac:c6
brgphy0 at bge0 phy 1: BCM5703 1000BASE-T media interface, rev. 2
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1 at pci2 dev 3 function 0: Broadcom BCM5703X Gigabit Ethernet
bge1: interrupting at ioapic1 pin 2 (irq 7)
bge1: ASIC BCM5703 A2 (0x1002), Ethernet address 00:09:3d:12:ac:c7
brgphy1 at bge1 phy 1: BCM5703 1000BASE-T media interface, rev. 2
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
mpt0 at pci2 dev 4 function 0: LSI Logic 53c1030 Ultra320 SCSI
mpt0: interrupting at ioapic1 pin 3 (irq 11)
scsibus0 at mpt0: 16 targets, 8 luns per target
aapic0 at pci0 dev 10 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
ppb2 at pci0 dev 11 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled
aapic1 at pci0 dev 11 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
pchb0 at pci0 dev 24 function 0
pchb0: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb1 at pci0 dev 24 function 1
pchb1: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb2 at pci0 dev 24 function 2
pchb2: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb3 at pci0 dev 24 function 3
pchb3: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
pchb4 at pci0 dev 25 function 0
pchb4: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb5 at pci0 dev 25 function 1
pchb5: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb6 at pci0 dev 25 function 2
pchb6: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb7 at pci0 dev 25 function 3
pchb7: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
pchb8 at pci0 dev 26 function 0
pchb8: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb9 at pci0 dev 26 function 1
pchb9: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb10 at pci0 dev 26 function 2
pchb10: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb11 at pci0 dev 26 function 3
pchb11: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
pchb12 at pci0 dev 27 function 0
pchb12: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb13 at pci0 dev 27 function 1
pchb13: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb14 at pci0 dev 27 function 2
pchb14: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb15 at pci0 dev 27 function 3
pchb15: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0 mux 1
wskbd0: connecting to wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
attimer0 at isa0 port 0x40-0x43: AT Timer
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
pcppi0: attached to attimer0
pci4 at mainbus0 bus 32
pci4: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
ppb3 at pci4 dev 1 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci5 at ppb3 bus 33
pci5: i/o space, memory space enabled
aapic2 at pci4 dev 1 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
ppb4 at pci4 dev 2 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci6 at ppb4 bus 37
pci6: i/o space, memory space enabled
aapic3 at pci4 dev 2 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
ppb5 at pci4 dev 3 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci7 at ppb5 bus 41
pci7: i/o space, memory space enabled
aapic4 at pci4 dev 3 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
ppb6 at pci4 dev 4 function 0: Advanced Micro Devices AMD8131 PCI-X Tunnel (rev. 0x12)
pci8 at ppb6 bus 45
pci8: i/o space, memory space enabled
aapic5 at pci4 dev 4 function 1: Advanced Micro Devices AMD8131 IO Apic (rev. 0x01)
ioapic0: enabling
ioapic1: enabling
ioapic2: enabling
ioapic3: enabling
ioapic4: enabling
ioapic5: enabling
ioapic6: enabling
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
raidattach: Asked for 8 units
Kernelized RAIDframe activated
scsibus0: waiting 2 seconds for devices to settle...
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 0: <MATSHITADVD-ROM SR-8178, , PY19> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(viaide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST373207LC, 0002> disk fixed
sd0: 70007 MB, 90774 cyl, 2 head, 789 sec, 512 bytes/sect x 143374744 sectors
sd0: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST373207LC, 0002> disk fixed
sd1: 70007 MB, 90774 cyl, 2 head, 789 sec, 512 bytes/sect x 143374744 sectors
sd1: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
ses0 at scsibus0 target 6 lun 0: <SDR, GEM318P, 1> processor fixed
ses0: SAF-TE Compliant Device
ses0: async, 8-bit transfers
Searching for RAID components...
Component on: sd0a: 143374681
   Row: 0 Column: 0 Num Rows: 1 Num Columns: 2
   Version: 2 Serial Number: 2005071501 Mod Counter: 294
   Clean: Yes Status: 0
   sectPerSU: 128 SUsPerPU: 1 SUsPerRU: 1
   RAID Level: 1  blocksize: 512 numBlocks: 143374592
   Autoconfig: Yes
   Contains root partition: Yes
   Last configured as: raid0
Component on: sd1a: 143374681
   Row: 0 Column: 1 Num Rows: 1 Num Columns: 2
   Version: 2 Serial Number: 2005071501 Mod Counter: 294
   Clean: Yes Status: 0
   sectPerSU: 128 SUsPerPU: 1 SUsPerRU: 1
   RAID Level: 1  blocksize: 512 numBlocks: 143374592
   Autoconfig: Yes
   Contains root partition: Yes
   Last configured as: raid0
Found: sd0a at 0
Found: sd1a at 1
RAID autoconfigure
Configuring raid0:
Starting autoconfiguration of RAID set...
Looking for 0 in autoconfig
Found: sd0a at 0
Looking for 1 in autoconfig
Found: sd1a at 1
raid0: allocating 20 buffers of 65536 bytes.
raid0: RAID Level 1
raid0: Components: /dev/sd0a /dev/sd1a
raid0: Total Sectors: 143374592 (70007 MB)
boot device: raid0
root on raid0a dumps on raid0b
mountroot: trying lfs...
mountroot: trying ffs...
root file system type: ffs
cpu1: CPU 1 running
init: copying out path `/sbin/init' 11
cpu2: CPU 2 running
cpu3: CPU 3 running
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)



>How-To-Repeat:

  configure and start building a kernel on a quad-opteron box.
  while building, observe that 'top' shows high 'system' usage for all
of the CPUs, and that 'vmstat -i 1' shows high interrupt rates for all
cpus. 
  Do a test './build.sh ... -j 4 ...' and observe that it takes longer
than one would really expect.

Alternately: 

  Run the High-Performance Linpack Benchmarks
(http://www.netlib.org/benchmark/hpl/) and observe the same high
'system' usage in top when a (large-memory (1GB per process x 4)
benchmark is exiting.  While each benchmark is running, 99% of CPU is
devoted to running the benchmark, but at process exit time is when the
'system' usage goes quite high (70-90%) for the duration of the
process exit.  The process exit can also take quite some time (minutes
for a normal exit, many minutes for a CTRL-C'ed kill..) 

>Fix:
No idea.  I'd be happy to test any potential fixes, however.