Subject: kern/24829: panic: postsig action on SMP system
To: None <gnats-bugs@gnats.netbsd.org>
From: None <jarle@uninett.no>
List: netbsd-bugs
Date: 03/17/2004 18:20:43
>Number: 24829
>Category: kern
>Synopsis: panic: postsig action on SMP system
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Mar 17 17:21:00 UTC 2004
>Closed-Date:
>Last-Modified:
>Originator: Jarle Greipsland
>Release: NetBSD 1.6ZK as of 2004-03-16
>Organization:
>Environment:
System: NetBSD rallar.urc.uninett.no 1.6ZK NetBSD 1.6ZK (RALLAR) #14: Tue Mar 16 16:41:54 CET 2004 he@rallar.urc.uninett.no:/usr/obj/sys/arch/i386/compile/RALLAR i386
Architecture: i386
Machine: i386
>Description:
I just discovered the recently commited modifications to the
LWP/POSIX-threads code, that introduced support for scheduling multiple
LWPs from the same process on multiple CPUs. I wanted to investigate this
feature, and built apache2, from pkgsrc, with the 'worker' multiprocessing
module, instead of the default 'prefork' module. I started apache with the
command:
# env PTHREAD_CONCURRENCY=4 /usr/pkg/etc/rc.d/apache forcestart
and used the default configuration file for the web server.
I also compiled http_load from pkgsrc, and ran it several times against a
few URLs for the Apache documentation. It all seemed very nice. I then
decided to stop the Apache, and used the command:
# /usr/pkg/etc/rc.d/apache forcestop
and the system panicked with the message:
panic: postsig action
Below, you will find:
o The console log and interaction with DDB
o The dmesg output from the post-panic reboot
o The configuration file
Stopped in pid 8461.42 (httpd) at netbsd:cpu_Debugger+0x4: leave
db{0}> trace
cpu_Debugger(54b,0,0,d2da63c8,80c1bc4) at netbsd:cpu_Debugger+0x4
panic(c06e00ec,ce83f32c,cf58bf0c,c035edaa,c07ea460) at netbsd:panic+0x121
postsig(f,cf58bf64,cf58bf5c,6,c07ea460) at netbsd:postsig+0x88
syscall_plain() at netbsd:syscall_plain+0xcc
--- syscall (number 8461) ---
0x484117d7:
db{0}> show reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0x1
esi 0x100
ebp 0xcf58be9c usb_all_tasks+0xed84a24
ebx 0x1
edx 0xf8000ef8
ecx 0x3fd
eax 0x1633
eip 0xc0410e5c cpu_Debugger+0x4
cs 0x8
eflags 0x202
esp 0xcf58be9c usb_all_tasks+0xed84a24
ss 0x10
netbsd:cpu_Debugger+0x4: leave
db{0}> mach cpuo 1
No such command
db{0}> mach cpu 1
using CPU 1
db{0}> trace
acquire(c07ea460,d43eae8c,400000,0,600) at netbsd:acquire+0x5e
_lockmgr(c07ea460,400002,0,c06dfb0b,54b) at netbsd:_lockmgr+0x4b0
_kernel_proc_lock(ce83f224,d43eaf80,d43eaf0c,c036068f,c07ea460) at netbsd:_kernel_proc_lock+0x39
sa_upcall_userret(ce83f224,d43eaf54,0,1,554) at netbsd:sa_upcall_userret+0x3c
trap() at netbsd:trap+0x45e
--- trap (number 6) ---
0:
db{0}> show reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0
esi 0xe004e
ebp 0xd43eae4c
ebx 0xc07ea460 kernel_lock
edx 0xf8000000
ecx 0
eax 0x480400
eip 0xc035e79e acquire+0x5e
cs 0x8
eflags 0x246
esp 0xcc2c3fbc usb_all_tasks+0xbabcb44
ss 0xcc2c0010 usb_all_tasks+0xbab8b98
netbsd:acquire+0x5e: jmp netbsd:acquire+0x49
db{0}> mach cpu 2
using CPU 2
db{0}> trace
acquire(c07ea460,d3e9fecc,400000,0,600) at netbsd:acquire+0x56
_lockmgr(c07ea460,400002,0,c06dfb0b,54b) at netbsd:_lockmgr+0x4b0
_kernel_proc_lock(ce1fdf00,48131000,0,1,d) at netbsd:_kernel_proc_lock+0x39
trap() at netbsd:trap+0x8a3
--- trap (number 6) ---
0x4810c7b1:
db{0}> show reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0
esi 0xe131c
ebp 0xd3e9fe8c
ebx 0xc07ea460 kernel_lock
edx 0xf8000000
ecx 0
eax 0x480400
eip 0xc035e796 acquire+0x56
cs 0x8
eflags 0x206
esp 0xcc2dffbc usb_all_tasks+0xbad8b44
ss 0xcc2d0010 usb_all_tasks+0xbac8b98
netbsd:acquire+0x56: jz netbsd:acquire+0xa8
db{0}> mach cpu 3
CPU 3 not configured
db{0}> mach cpu 4
using CPU 4
db{0}> trace
acquire(c07ea460,d26adebc,400000,0,600) at netbsd:acquire+0x58
_lockmgr(c07ea460,400002,0,c06dfb0b,54b) at netbsd:_lockmgr+0x4b0
_kernel_proc_lock(cefdf748,0,1,0,0) at netbsd:_kernel_proc_lock+0x39
sa_upcall_userret(cefdf748,d26adf64,d26adf5c,25,c07ea460) at netbsd:sa_upcall_userret+0x3c
syscall_plain() at netbsd:syscall_plain+0x105
--- syscall (number 0) ---
0x4839dc4b:
db{0}> show reg
ds 0x10
es 0x10
fs 0x30
gs 0x10
edi 0
esi 0xdf93a
ebp 0xd26ade7c
ebx 0xc07ea460 kernel_lock
edx 0xf8000000
ecx 0
eax 0x480400
eip 0xc035e798 acquire+0x58
cs 0x8
eflags 0x206
esp 0xcc2a7fbc usb_all_tasks+0xbaa0b44
ss 0xcc2a0010 usb_all_tasks+0xba98b98
netbsd:acquire+0x58: testl %edi,%edi
db{0}> ps
PID PPID PGRP UID S FLAGS LWPS COMMAND WAIT
*8461 16703 16703 1003 2 0x500 5 httpd *
19314 16703 16703 1003 2 0x500 4 httpd *
5837 16703 16703 1003 2 0x2500 42 httpd *
12664 16703 16703 1003 2 0x100 1 httpd
16703 1 16703 0 2 0 1 httpd select
6762 26956 6762 1001 2 0x4003 1 bash ttyin
26956 27604 27604 1001 2 0x4100 1 xterm select
27604 3005 27604 1001 2 0x4000 1 csh pause
3005 29253 29253 1001 2 0x100 1 sshd select
29253 307 29253 0 2 0x100 1 sshd netio
15944 20904 15944 1000 2 0x4102 1 systat ttyin
12830 2170 12830 0 2 0x4002 1 tcsh ttyin
2170 15281 2170 1000 2 0x4003 1 csh pause
15281 26039 26039 1000 2 0x4100 1 xterm select
26039 6470 26039 1000 2 0x4000 1 csh pause
6470 6180 6180 1000 2 0x100 1 sshd select
20904 4163 20904 1000 2 0x4003 1 csh pause
6180 307 6180 0 2 0x100 1 sshd netio
4163 20735 20735 1000 2 0x4100 1 xterm select
20735 18119 20735 1000 2 0x4000 1 csh pause
18119 11434 11434 1000 2 0x100 1 sshd select
11434 307 11434 0 2 0x100 1 sshd netio
28698 15092 28698 0 2 0x5002 1 tcsh ttyin
15092 14583 15092 1001 2 0x4002 1 bash wait
14583 13884 13884 1001 2 0x100 1 sshd select
13884 307 13884 0 2 0x101 1 sshd netio
29240 405 29240 0 2 0x4002 1 tail kqread
405 432 405 0 2 0x4002 1 tcsh pause
432 399 432 1000 2 0x4003 1 csh pause
399 396 396 1000 2 0x4100 1 xterm select
396 395 396 1000 2 0x4000 1 csh pause
395 105 105 1000 2 0x100 1 sshd select
105 307 105 0 2 0x100 1 sshd netio
104 98 366 1000 2 0x4000 1 setiathome
103 100 363 1000 2 0x4000 1 setiathome
102 373 364 1000 2 0x4000 1 setiathome
101 97 348 1000 2 0x4000 1 setiathome
100 1 363 1000 2 0x4000 1 sh wait
98 1 366 1000 2 0x4000 1 sh wait
97 1 348 1000 2 0x4000 1 sh wait
373 1 364 1000 2 0x4000 1 sh wait
367 1 367 0 2 0x4002 1 getty ttyin
365 1 365 0 2 0x4002 1 getty ttyin
349 1 349 0 2 0x4002 1 getty ttyin
351 1 351 0 2 0x4002 1 getty ttyin
344 336 336 0 2 0 1 cron piperd
342 336 336 0 2 0 1 cron piperd
346 336 336 0 2 0 1 cron piperd
345 336 336 0 2 0 1 cron piperd
336 1 336 0 2 0 1 cron nanosle
276 1 276 0 2 0 1 inetd kqread
307 1 307 0 2 0 1 sshd select
273 1 273 15 2 0x100 1 ntpd pause
187 1 187 0 2 0 1 mount_mfs mfsidl
146 1 146 0 2 0 1 syslogd poll
9 0 0 0 2 0x20200 1 aiodoned aiodone
8 0 0 0 2 0x20200 1 ioflush syncer
7 0 0 0 2 0x20200 1 pagedaemon pgdaemo
6 0 0 0 2 0x20200 1 lfs_writer lfswrit
5 0 0 0 2 0x20200 1 scsibus1 sccomp
4 0 0 0 2 0x20200 1 scsibus0 sccomp
3 0 0 0 2 0x20200 1 amr0 amrwdog
2 0 0 0 2 0x20200 1 cryptoret crypto_
1 0 1 0 2 0x4000 1 init wait
0 -1 0 0 2 0x20200 1 swapper schedul
363 345 363 1000 5 0x6000 0 sh *
366 346 366 1000 5 0x6000 0 sh *
348 344 348 1000 5 0x6000 0 sh *
364 342 364 1000 5 0x6000 0 sh *
db{0}> reboot 4
rebooting...
** NetBSD/i386 BIOS Boot, Revision 2.16
** (he@stegg.urc.uninett.no, Wed Apr 23 16:58:48 CEST 2003)
** Memory: 639/1047552 k
Press return to boot now, any other key for boot menu
booting hd0a:netbsd - starting in 0
6714900+147748+500572 [368624+329747]=0x7b1f10
BIOS CFG: Model-SubM-Rev: fc-01-00, 0x74<EBDA,KBDINT,RTC,IC2>
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 1.6ZK (RALLAR) #14: Tue Mar 16 16:41:54 CET 2004
he@rallar.urc.uninett.no:/usr/obj/sys/arch/i386/compile/RALLAR
total memory = 1023 MB
avail memory = 983 MB
BIOS32 rev. 0 found at 0xfd070
mainbus0 (root)
mainbus0: Intel MP Specification (Version 1.4) (INTEL ALDER )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Pentium Pro (686-class), 198.96 MHz, id 0x619
cpu0: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features fbff<PGE,MCA,CMOV>
cpu0: I-cache 8 KB 32b/line 4-way, D-cache 8 KB 32b/line 2-way
cpu0: L2 cache 512 KB 32b/line 4-way
cpu0: ITLB 32 4 KB entries 4-way, 2 4 MB entries fully associative
cpu0: DTLB 64 4 KB entries 4-way, 8 4 MB entries 4-way
cpu0: calibrating local timer
cpu0: apic clock running at 66 MHz
cpu0: 32 page colors
cpu1 at mainbus0: apid 4 (application processor)
cpu1: starting
cpu1: Intel Pentium Pro (686-class), 198.95 MHz, id 0x619
cpu1: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features fbff<PGE,MCA,CMOV>
cpu1: I-cache 8 KB 32b/line 4-way, D-cache 8 KB 32b/line 2-way
cpu1: L2 cache 512 KB 32b/line 4-way
cpu1: ITLB 32 4 KB entries 4-way, 2 4 MB entries fully associative
cpu1: DTLB 64 4 KB entries 4-way, 8 4 MB entries 4-way
cpu2 at mainbus0: apid 1 (application processor)
cpu2: starting
cpu2: Intel Pentium Pro (686-class), 198.95 MHz, id 0x619
cpu2: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu2: features fbff<PGE,MCA,CMOV>
cpu2: I-cache 8 KB 32b/line 4-way, D-cache 8 KB 32b/line 2-way
cpu2: L2 cache 512 KB 32b/line 4-way
cpu2: ITLB 32 4 KB entries 4-way, 2 4 MB entries fully associative
cpu2: DTLB 64 4 KB entries 4-way, 8 4 MB entries 4-way
cpu3 at mainbus0: apid 2 (application processor)
cpu3: starting
cpu3: Intel Pentium Pro (686-class), 198.95 MHz, id 0x619
cpu3: features fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu3: features fbff<PGE,MCA,CMOV>
cpu3: I-cache 8 KB 32b/line 4-way, D-cache 8 KB 32b/line 2-way
cpu3: L2 cache 512 KB 32b/line 4-way
cpu3: ITLB 32 4 KB entries 4-way, 2 4 MB entries fully associative
cpu3: DTLB 64 4 KB entries 4-way, 8 4 MB entries 4-way
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 18 is type EISA
ioapic0 at mainbus0 apid 14 (I/O APIC)
ioapic0: pa 0xfec00000, version 11, 16 pins
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
fxp0 at pci0 dev 11 function 0: Intel i82557 Ethernet, rev 2
fxp0: interrupting at ioapic0 pin 10 (irq 10)
fxp0: Ethernet address 00:a0:c9:4c:b0:0c
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pceb0 at pci0 dev 14 function 0
pceb0: Intel 82375EB/SB PCI-EISA Bridge (PCEB) (rev. 0x15)
Intel product 0x0008 (undefined subclass 0x00) at pci0 dev 15 function 0 not configured
Intel 82451KX/GX Memory Controller (MC) (RAM memory, revision 0x05) at pci0 dev 20 function 0 not configured
pchb0 at pci0 dev 25 function 0
pchb0: Intel 82454KX/GX PCI Bridge (PB) (rev. 0x06)
pchb0: Compatibility PB (bus 0)
pchb1 at pci0 dev 26 function 0
pchb1: Intel 82454KX/GX PCI Bridge (PB) (rev. 0x06)
pchb1: Auxiliary PB (bus 1)
pci1 at pchb1 bus 1
pci1: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
amr0 at pci1 dev 10 function 0: AMI RAID <Series 428>
amr0: interrupting at ioapic0 pin 11 (irq 11)
amr0: firmware <U.84>, BIOS <1.63>, 32MB RAM
ld0 at amr0 unit 0: RAID 0, optimal
ld0: 69356 MB, 8841 cyl, 255 head, 63 sec, 512 bytes/sect x 142041088 sectors
ahc1 at pci1 dev 11 function 0: Adaptec aic7880 Ultra SCSI adapter
ahc1: interrupting at ioapic0 pin 9 (irq 9)
ahc1: Using left over BIOS settings
ahc1: aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
scsibus0 at ahc1: 16 targets, 8 luns per target
ahc2 at pci1 dev 12 function 0: Adaptec aic7880 Ultra SCSI adapter
ahc2: interrupting at ioapic0 pin 5 (irq 5)
ahc2: Using left over BIOS settings
ahc2: aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
scsibus1 at ahc2: 16 targets, 8 luns per target
eisa0 at pceb0
eisa0: can't map I/O space for slot 14
eisa0: can't map I/O space for slot 15
isa0 at pceb0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
vga0 at isa0 port 0x3b0-0x3df iomem 0xa0000-0xbffff
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
isapnp0: no ISA Plug 'n Play devices found
ioapic0: enabling
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST39102L CLAR09, 3504> disk fixed
sd0: 8625 MB, 6962 cyl, 12 head, 211 sec, 512 bytes/sect x 17664229 sectors
sd0: sync (50.00ns offset 8), 16-bit (40.000MB/s) transfers, tagged queueing
cd0 at scsibus1 target 5 lun 0: <NEC, CD-ROM DRIVE:464, 1.05> cdrom removable
cd0: sync (50.00ns offset 15), 8-bit (20.000MB/s) transfers
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
cpu2: CPU 1 running
cpu1: CPU 4 running
cpu3: CPU 2 running
Wed Mar 17 14:26:54 GMT 2004
The RALLAR configuration file:
------------------------------------------------------------
# $NetBSD: GENERIC.MP,v 1.4 2003/03/05 10:28:00 grant Exp $
#
# GENERIC.MP -- Generic, plus enable Multiprocessor support.
#
include "arch/i386/conf/GENERIC"
options MULTIPROCESSOR
options COM_MPLOCK # com MP locking; REQUIRED on MP i386
options APM_NO_IDLE
options DIAGNOSTIC # STRONGLY RECOMMENDED
options LOCKDEBUG
#options MPDEBUG
#options MPVERBOSE
#options DEBUG
options MPBIOS # configure CPUs and APICs using MPBIOS
#options MPACPI # configure CPUs and APICs using ACPI
# (acpi at mainbus must also be enabled)
ioapic* at mainbus? apid ?
------------------------------------------------------------
The included GENERIC config file is version 1.595 from the NetBSD CVS
repository.
>How-To-Repeat:
I have not tried to repeat this yet, but presumably the panic could be
provoked again by repating the start+load+stop sequence described above.
>Fix:
Unknown.
>Release-Note:
>Audit-Trail:
>Unformatted: