Subject: Attempting to stop lock up when out of memory
To: None <tech-kern@NetBSD.org>
From: Jeremy C. Reed <reed@reedmedia.net>
List: tech-kern
Date: 07/21/2005 15:36:39
(Please carbon copy me on replies.)
For a few years, my i386 system running 1.6.x and then 2.0.2 often locks
up when I am out of memory. pinging still works for awhile and ls listing
from nfs works extremely slow. But waiting a lot longer even can't ping
system. Log files show nothing.
It would be nice if there were some sane defaults for memory resource
limits and also if the system could be kept available.
My system has:
total memory = 127 MB
avail memory = 117 MB
And it has 192 MB of swap.
The following are all defaults.
As a regular user my limits are:
proc.curproc.rlimit.datasize.soft = 134217728 (128 MB)
proc.curproc.rlimit.datasize.hard = 1073741824 (1 GB)
proc.curproc.rlimit.stacksize.soft = 2097152 (2 MB)
proc.curproc.rlimit.stacksize.hard = 33554432 (32 MB)
proc.curproc.rlimit.memoryuse.soft = 122421248 (116 MB)
proc.curproc.rlimit.memoryuse.hard = 122421248
proc.curproc.rlimit.memorylocked.soft = 40807082 (38 MB)
proc.curproc.rlimit.memorylocked.hard = 122421248 (116 MB)
Any my shell tells me my same limits:
data seg size (kbytes, -d) 131072 -- soft
data seg size (kbytes, -d) 1048576
max locked memory (kbytes, -l) 39850 -- soft
max locked memory (kbytes, -l) 119552
max memory size (kbytes, -m) 119552 -- soft
max memory size (kbytes, -m) 119552
stack size (kbytes, -s) 2048 -- soft
stack size (kbytes, -s) 32768
virtual memory (kbytes, -v) 133120 -- soft
virtual memory (kbytes, -v) 1081344 (1056 MB)
As root these limits were identical.
On a second system with:
total memory = 93820 KB
avail memory = 84012 KB
with a 25704 KBytes swap, I have the following:
proc.curproc.rlimit.datasize.soft = 134217728
proc.curproc.rlimit.datasize.hard = 1073741824
proc.curproc.rlimit.stacksize.soft = 2097152
proc.curproc.rlimit.stacksize.hard = 33554432
proc.curproc.rlimit.memoryuse.soft = 85405696 (81 MB)
proc.curproc.rlimit.memoryuse.hard = 85405696
proc.curproc.rlimit.memorylocked.soft = 28468565 (27 MB)
proc.curproc.rlimit.memorylocked.hard = 85405696
data(kbytes) 131072
stack(kbytes) 2048
lockedmem(kbytes) 27801
memory(kbytes) 83404
I didn't know what some of this meant, but I found that login.conf
describes memorylocked as "Maximum locked in core memory size limit" and
memoryuse as "Maximum in core memoryuse size limit". csh documents
memoryuse as "the maximum size (in bytes) to which a process's resident
set size (RSS) may grow" and memorylocked as "The maximum size (in bytes)
which a process may lock into memory using the mlock(2) function."
What can be done so the default out-of-the-box configurations won't allow
the system to become locked up or unavailable when out of memory?
It seems like default syslog.conf is configured fine, but I don't see any
messages logged from kernel when running out of memory. I am running X, so
don't see on console either. And maybe syslogd is already dead -- I can't
tell. And I don't see any "killed: out of swap" either.
(Yes, I know I need more swap and/or more physical memory, but I am hoping
it not become unavailable when I hit these problems.)
Maybe sys/arch/i386/i386/trap.c (or for other platforms) can have the
kernel tsleep for a moment and retest for the ENOMEM before it kills the
process when out of swap. Any thoughts about that?
Or maybe it can be made to kill off latest processes first?
But even the "out of swap" does not explain to me why pinging finally
stops working too. Any ideas?
(Please CC me on replies.)
Jeremy C. Reed
BSD News, BSD tutorials, BSD links
http://www.bsdnewsletter.com/