Subject: ARP lockup (arp_drain: locked; punting)
To: None <tech-net@NetBSD.org>
From: Edgar =?unknown-8bit?B?RnXf?= <efnbl05@bn2.maus.net>
List: tech-net
Date: 03/24/2006 11:46:21
We're experiencing ARP lockups on one of our servers since upgrading
from 2.0.2 to 3.0.
The only uncommon thing about the machine in question is that it has two
network interfaces connected to the same ``wire''. We run two IP nets
(131.220.120/24 and 131.220.146/24) on the same ethernet (until I find the
time to partition the switches) and this was the way to make DHCP work.
The lockups occur at 04:48; I can't find anything special going on at
that time apart from our FAI clients starting a softupdate at 04:30;
that shouldn't put any heavy load on that particular server.
I get ``arp_drain: locked; punting'' in syslog and from that point on,
ARP seems to stop functioning. I can ping -n the machine's own IP,
but nothing else.
The interfaces in question are epic's, so this can't be the nfe memory
leak discussed in current-users.
The machine acts as a DHCP server, HTTP/FTP server and HTTP/FTP/RTSP proxy.
It also runs some internal list using Mailman.
vmstat -v -m -W (in the locked-up state) says:
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1658 390 100741 1280 0
32 311 201 54880 640 0
64 704 2048 49545 320 720
128 242 110 16648 160 0
256 358 26 10956 80 0
512 80 40 33893 40 0
1024 264 248 26781 20 11592
2048 34 28 3364 10 3100
4096 54 21 280 5 206
8192 20 0 37 5 4
16384 25 0 37 5 0
32768 20 0 46 5 0
65536 5 0 5 5 0
Memory usage type by bucket size
Size Type(s)
16 USB, VM swap, UVM amap, ifaddr, namecache, soname, pipe, in_multi,
ip_moptions, routetbl, temp, pcb, devbuf, exec, kernfs mount,
RAIDframe, xform
32 USB, AGP, VM swap, UVM amap, ifmedia, ether_multi, ifaddr, soname,
sysctldata, proc, fragtbl, routetbl, temp, pcb, devbuf, exec, kevent,
RAIDframe, xform
64 USB, UVM amap, vnodes, prop, in_multi, routetbl, temp, pcb, DMA map,
devbuf, MFS node, NFS diroff, NFS req, NDP
128 USB device, USB, AGP, VM swap, UVM amap, ifaddr, soname, ttys, prop,
sysctlnode, routetbl, temp, devbuf, kevent, inodedep, RAIDframe,
key mgmt, NDP
256 USB, UVM amap, ifaddr, vnodes, subproc, proc, routetbl, temp, DMA map,
devbuf, exec, kevent, file desc, newblk, NFS diroff
512 USB device, VM map, UVM amap, ifaddr, proc, UFS mount, temp, pcb,
devbuf, ioctlops, file desc, NFS mount, RAIDframe, crypto
1024 VM page, UVM aobj, UVM amap, ttys, sysctlnode, temp, devbuf, exec,
kevent, NQNFS Lease, RAIDframe, key mgmt
2048 UVM amap, ifaddr, sysctlnode, in_multi, UFS mount, temp, devbuf, exec,
RAIDframe
4096 UVM amap, mount, namecache, sysctlnode, temp, devbuf, RAIDframe
8192 UVM amap, sysctlnode, UFS mount, pagedep, RAIDframe
16384 UVM amap, sysctlnode, RAIDframe
32768 UVM amap, namecache, UFS mount, devbuf, inodedep, NFS node,
ISOFS mount, RAIDframe
65536 UVM amap, temp, devbuf
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
USB device 4 2K 2K 58752K 4 0 0 128,512
USB 16 2K 2K 58752K 16 0 0 16,32,64,128,256
AGP 2 1K 1K 58752K 2 0 0 32,128
VM swap 5 1K 1K 58752K 5 0 0 16,32,128
VM page 1 1K 1K 58752K 1 0 0 1024
VM map 4 2K 2K 58752K 4 0 0 512
UVM aobj 1 1K 1K 58752K 1 0 0 1024
UVM amap 2401 464K 562K 58752K 192935 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
ifmedia 12 1K 1K 58752K 12 0 0 32
ether_multi 10 1K 1K 58752K 10 0 0 32
ifaddr 22 7K 7K 58752K 22 0 0 16,32,128,256,512,2048
mount 10 40K 40K 58752K 11 0 0 4096
vnodes 43 7K 126K 58752K 9481 0 0 64,256
namecache 4 69K 69K 58752K 4 0 0 16,4096,32768
soname 4 1K 12K 58752K 2976 0 0 16,32,128
ttys 36 23K 23K 58752K 36 0 0 128,1024
pipe 0 0K 1K 58752K 13 0 0 16
prop 2 1K 1K 58752K 2 0 0 64,128
sysctldata 0 0K 1K 58752K 1 0 0 32
sysctlnode 52 88K 89K 58752K 76 0 0 128,1024,2048,4096,8192,16384
subproc 4 1K 3K 58752K 62 0 0 256
proc 8 1K 2K 58752K 16 0 0 32,256,512
in_multi 34 4K 4K 58752K 34 0 0 16,64,2048
ip_moptions 1 1K 1K 58752K 1 0 0 16
UFS mount 21 75K 75K 58752K 21 0 0 512,2048,8192,32768
fragtbl 0 0K 1K 58752K 77 0 0 32
routetbl 101 7K 12K 58752K 633 0 0 16,32,64,128,256
temp 12 58K 62K 58752K 53640 0 0 16,32,64,128,256,512,1024,2048,4096,65536
pcb 22 4K 13K 58752K 3069 0 0 16,32,64,512
DMA map 400 74K 74K 58752K 400 0 0 64,256
devbuf 238 262K 262K 58752K 660 0 0 16,32,64,128,256,512,1024,2048,4096,32768,65536
exec 2 1K 7K 58752K 6377 0 0 16,32,256,1024,2048
kevent 7 3K 3K 58752K 4212 0 0 32,128,256,1024
ioctlops 0 0K 1K 58752K 31 0 0 512
file desc 1 1K 5K 58752K 300 0 0 256,512
MFS node 1 1K 1K 58752K 1 0 0 64
newblk 1 1K 1K 58752K 1 0 0 256
inodedep 1 32K 33K 58752K 4 0 0 128,32768
pagedep 1 8K 8K 58752K 1 0 0 8192
NFS mount 3 2K 2K 58752K 3 0 0 512
NFS diroff 31 3K 4K 58752K 40 0 0 64,256
NFS req 0 0K 1K 58752K 362 0 0 64
NQNFS Lease 1 1K 1K 58752K 1 0 0 1024
NFS node 1 32K 32K 58752K 1 0 0 32768
kernfs mount 1 1K 1K 58752K 1 0 0 16
ISOFS mount 1 32K 32K 58752K 1 0 0 32768
RAIDframe 65 690K 720K 58752K 76 0 0 16,32,128,512,1024,2048,4096,8192,16384,32768
key mgmt 181 175K 427K 58752K 21542 0 0 128,1024
NDP 6 1K 1K 58752K 8 0 0 64,128
crypto 1 1K 1K 58752K 1 0 0 512
xform 0 0K 1K 58752K 28 0 0 16,32
Memory totals: In Use Free Requests
2164K 569K 297216
Memory resource pool statistics
Name Size Requests Fail Releases InUse Avail Pgreq Pgrel Npage PageSz Hiwat Minpg Maxpg Idle Flags Util
phpool-0 40 1286 0 525 761 47 8 0 8 4096 8 0 inf 0 0x040 92.9%
phpool-2 44 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
phpool-4 44 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
phpool-8 48 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
phpool-16 56 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
phpool-32 72 50 0 0 50 6 1 0 1 4096 1 0 inf 0 0x040 87.9%
phpool-64 104 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
phpool-128 168 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
pcgpool 140 67 0 46 21 7 1 0 1 4096 1 0 inf 0 0x040 71.8%
pmappl 76 3523 0 3494 29 24 1 0 1 4096 1 0 inf 0 0x040 53.8%
pdppl 4096 62 0 24 38 4 44 2 42 4096 42 0 inf 4 0x000 90.5%
cache 0xc0363560: allocfrom 0xc067e08c freeto 0xc067e08c
hits 3461 misses 62 ngroups 1 nitems 9
kvakernel 4096 1289 0 142 1147 5 36 0 36 131072 36 0 0 0 0x600 99.6%
kvakmem 4096 548 0 104 444 4 14 0 14 131072 14 0 0 0 0x600 99.1%
igmppl 20 2 0 0 2 200 1 0 1 4096 1 0 inf 0 0x040 1.0%
inpcbpl 108 21508 0 21344 164 206 16 6 10 4096 12 0 inf 0 0x040 43.2%
inmltpl 32 3 0 0 3 123 1 0 1 4096 1 0 inf 0 0x040 2.3%
ipqepl 44 442 0 442 0 0 2 2 0 4096 1 0 inf 0 0x040 ---
tcpipqepl 44 18819 0 18819 0 0 1 1 0 4096 1 0 inf 0 0x040 ---
synpl 184 7003 0 7003 0 0 2 2 0 4096 1 0 inf 0 0x040 ---
sackholepl 20 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
tcpcbpl 360 18387 0 18236 151 201 54 22 32 4096 37 0 inf 4 0x040 41.5%
in6pcbpl 156 26 0 16 10 16 1 0 1 4096 1 0 inf 0 0x040 38.1%
cd9660nopl 160 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
nfsnodepl 308 35 0 27 8 18 2 0 2 4096 2 0 inf 1 0x040 30.1%
nfsvapl 108 35 0 27 8 29 1 0 1 4096 1 0 inf 0 0x040 21.1%
sdpcpool 124 19293 0 19292 1 351 29 18 11 4096 16 0 inf 10 0x040 0.3%
pagedeppl 68 1438 0 1437 1 58 2 1 1 4096 1 0 inf 0 0x040 1.7%
inodedeppl 88 15335 0 15334 1 321 24 17 7 4096 15 0 inf 6 0x040 0.3%
newblkpl 36 37156 0 37156 0 112 1 0 1 4096 1 0 inf 1 0x040 0.0%
bmsafemappl 36 2561 0 2560 1 111 1 0 1 4096 1 0 inf 0 0x040 0.9%
allocdirectpl 80 31863 0 31862 1 299 19 13 6 4096 11 0 inf 5 0x040 0.3%
indirdeppl 32 368 0 368 0 126 1 0 1 4096 1 0 inf 1 0x040 0.0%
allocindirpl 64 5293 0 5293 0 441 19 12 7 4096 7 0 inf 7 0x040 0.0%
freefragpl 40 14510 0 14510 0 202 4 2 2 4096 2 0 inf 2 0x040 0.0%
freeblkspl 172 6884 0 6884 0 138 22 16 6 4096 10 0 inf 6 0x040 0.0%
freefilepl 36 6918 0 6918 0 224 5 3 2 4096 3 0 inf 2 0x040 0.0%
diraddpl 36 8584 0 8583 1 223 4 2 2 4096 3 0 inf 1 0x040 0.4%
mkdirpl 32 2 0 2 0 126 1 0 1 4096 1 0 inf 1 0x040 0.0%
dirrempl 36 7180 0 7180 0 224 5 3 2 4096 2 0 inf 2 0x040 0.0%
newdirblkpl 20 1 0 1 0 202 1 0 1 4096 1 0 inf 1 0x040 0.0%
ffsinopl 196 248834 0 242744 6090 270 318 0 318 4096 318 0 inf 0 0x040 91.6%
dino1pl 128 248834 0 242744 6090 265 205 0 205 4096 205 0 inf 0 0x040 92.8%
dino2pl 256 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x000 ---
ufsdirpl 264 8554 0 8554 0 15 2 1 1 4096 1 0 inf 1 0x040 0.0%
rndsample 528 2324 0 2321 3 4 1 0 1 4096 1 0 inf 0 0x040 38.7%
filepl 56 697102 0 696911 191 97 6 2 4 4096 4 0 inf 0 0x040 65.3%
cwdipl 16 3524 0 3494 30 223 1 0 1 4096 1 0 inf 0 0x040 11.7%
fdescpl 284 3524 0 3494 30 12 3 0 3 4096 3 0 inf 0 0x040 69.3%
kqueuepl 260 2 0 0 2 13 1 0 1 4096 1 0 inf 0 0x040 12.7%
knotepl 72 13 0 0 13 43 1 0 1 4096 1 0 inf 0 0x040 22.9%
ktepl 56 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
procpl 408 3539 0 3494 45 15 8 2 6 4096 6 0 inf 1 0x000 74.7%
lwppl 132 3539 0 3494 45 15 2 0 2 4096 2 0 inf 0 0x040 72.5%
lwpucpl 776 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
pgrppl 16 221 0 201 20 233 1 0 1 4096 1 0 inf 0 0x040 7.8%
pcredpl 24 3539 0 3494 45 124 1 0 1 4096 1 0 inf 0 0x040 26.4%
plimitpl 176 50 0 45 5 18 1 0 1 4096 1 0 inf 0 0x040 21.5%
pstatspl 224 3539 0 3494 45 27 4 0 4 4096 4 0 inf 1 0x040 61.5%
rusgepl 72 3494 0 3494 0 56 3 2 1 4096 1 0 inf 1 0x040 0.0%
raspl 20 33 0 24 9 193 1 0 1 4096 1 0 inf 0 0x040 4.4%
sadatapl 40 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
saupcpl 1612 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
sastackpl 24 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
savppl 52 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
ptimerpl 124 62 0 58 4 28 1 0 1 4096 1 0 inf 0 0x040 12.1%
sessionpl 40 186 0 169 17 84 1 0 1 4096 1 0 inf 0 0x040 16.6%
credpl 84 1137 0 1120 17 79 2 0 2 4096 2 0 inf 0 0x040 17.4%
siginfo 128 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
ksiginfo 44 2 0 2 0 92 1 0 1 4096 1 0 inf 1 0x040 0.0%
strprocpl 104 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
strpolpl 28 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
strmsgpl 300 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
pipepl 144 4654 0 4646 8 20 1 0 1 4096 1 0 inf 0 0x040 28.1%
shmmp 12 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
ttypl 264 7 0 0 7 8 1 0 1 4096 1 0 inf 0 0x040 45.1%
sockpl 216 24592 0 24272 320 184 43 15 28 4096 31 0 inf 3 0x040 60.3%
ncachepl 84 220970 0 214864 6106 278 155 22 133 4096 133 0 inf 0 0x040 94.2%
lockfpl 52 964 0 963 1 77 1 0 1 4096 1 0 inf 0 0x040 1.3%
vnodepl 168 13096 0 6986 6110 250 265 0 265 4096 265 0 inf 0 0x040 94.6%
rtentpl 132 290 0 242 48 72 4 0 4 4096 4 0 inf 0 0x040 38.7%
rttmrpl 32 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
amappl 40 59293 0 58500 793 217 12 2 10 4096 10 0 inf 0 0x040 77.4%
uaoeltpl 84 8 0 3 5 43 1 0 1 4096 1 0 inf 0 0x040 10.3%
aobjpl 52 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
vmsppl 224 3523 0 3494 29 25 5 2 3 4096 3 0 inf 1 0x040 52.9%
vmmpepl 88 293077 0 291630 1447 393 48 8 40 4096 40 0 inf 3 0x040 77.7%
swp vnx 20 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
swp vnd 128 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
ataspl 52 146592 0 146592 0 78 1 0 1 4096 1 0 inf 1 0x040 0.0%
bufpl 124 230221 0 227531 2690 3326 189 1 188 4096 188 0 inf 7 0x040 43.3%
buf1k 1024 14191 0 12036 2155 725 185 140 45 65536 78 1 1 0 0x000 74.8%
buf2k 2048 332 0 307 25 7 7 6 1 65536 4 1 1 0 0x000 78.1%
buf4k 4096 193 0 174 19 13 5 3 2 65536 3 1 1 0 0x000 59.4%
buf8k 8192 7973 0 7482 491 85 436 364 72 65536 229 1 1 0 0x000 85.2%
buf16k 16384 0 0 0 0 4 1 0 1 65536 1 1 1 1 0x000 0.0%
buf32k 32768 0 0 0 0 2 1 0 1 65536 1 1 1 1 0x000 0.0%
buf64k 65536 0 0 0 0 1 1 0 1 65536 1 1 1 1 0x000 0.0%
mbpl 256 1842 0 449 1393 31 90 1 89 4096 89 1 inf 1 0x000 97.8%
cache 0xc03605c0: allocfrom 0xc067e460 freeto 0xc067e460
hits 2079559 misses 1842 ngroups 13 nitems 137
mclpl 2048 1100 100452 100 1000 0 507 7 500 4096 500 4 512 0 0x004 100.0%
cache 0xc0360280: allocfrom 0x0 freeto 0xc067e71c
hits 778353 misses 101553 ngroups 6 nitems 0
pnbufpl 1024 12 0 8 4 0 1 0 1 4096 1 0 inf 0 0x000 100.0%
cache 0xc0360bc0: allocfrom 0xc067e000 freeto 0xc067e000
hits 954678 misses 12 ngroups 1 nitems 4
extent 20 12524 0 11203 1321 295 8 0 8 4096 8 0 inf 0 0x040 80.6%
ccdpl 140 0 0 0 0 0 0 0 0 4096 0 0 inf 0 0x040 ---
raidpl 132 146532 0 146532 0 120 8 4 4 4096 4 2 4 4 0x040 0.0%
sigapl 2052 3524 0 3494 30 12 54 12 42 4096 42 0 inf 12 0x040 35.8%
rf_alloclist_pl 168 99305 0 99304 1 95 6 2 4 4096 4 3 11 3 0x040 1.0%
rf_asmhdr_pl 16 88741 0 88741 0 253 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_asm_pl 288 99304 0 99304 0 98 11 4 7 4096 7 5 14 7 0x040 0.0%
rf_asmhle_pl 8 0 0 0 0 507 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_pda_pl 40 198608 0 198608 0 202 4 2 2 4096 2 1 2 2 0x040 0.0%
rf_fss_pl 20 0 0 0 0 202 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_vfple_pl 8 0 0 0 0 507 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_vple_pl 8 30 0 0 30 477 1 0 1 4096 1 1 2 0 0x040 5.9%
rf_revent_pl 16 0 0 0 0 253 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_callbackpl 20 0 0 0 0 202 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_rad_pl 328 88741 0 88741 0 48 6 2 4 4096 4 3 11 4 0x040 0.0%
rf_stripelock_pl 28 99304 0 99304 0 144 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_mcpair_pl 12 0 0 0 0 338 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_dagnode_pl 444 444444 0 444444 0 279 61 30 31 4096 31 15 57 31 0x040 0.0%
rf_dagh_pl 76 99304 0 99304 0 106 4 2 2 4096 2 1 3 2 0x040 0.0%
rf_daglist_pl 236 99304 0 99304 0 68 8 4 4 4096 4 2 8 4 0x040 0.0%
rf_dagpcache_pl 712 0 0 0 0 10 2 0 2 4096 2 2 26 2 0x040 0.0%
rf_funclist_pl 8 99304 0 99304 0 507 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_reconbuffer_pl 76 0 0 0 0 53 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_dqd_pl 116 146532 0 146532 0 136 8 4 4 4096 4 2 8 4 0x040 0.0%
raidpsspl 128 0 0 0 0 31 1 0 1 4096 1 1 2 1 0x040 0.0%
In use 12968K, total allocated 16312K; utilization 79.5%
Thanks for any help.