Well my server ran a bit longer and handled some fair workload
yesterday, but it still wedged overnight. The most likely first-hung
process was a "find" started by one of the nightly cron jobs.
That was with "options BUFCACHE" left at the default of 15%, but with
vm.bufmem manually set back down to 10%.
I may be wrong about this, but IIUC the manual setting won't free up and
re-allocate all of the buffer memory tables, etc., just restrict their
use -- BUFCACHE has to be set lower at boot to actually avoid allocating
more space than needed. I'm about to reboot another kernel with
BUFCACHE set to 5% and we'll see how it does.
At Thu, 18 Jun 2009 14:04:17 -0400 (EDT), der Mouse
<mouse%Rodents-Montreal.ORG@localhost> wrote:
Subject: Re: more on the "vmmapva" kernel deadlocks on NetBSD-4
>
> > Some of you may remember my PR#s 38019 and 38246.
>
> I didn't, but thank you. This sounds suspiciously like a symptom I see
> on sparkle, my most central machine: I need to reboot every
> three-to-five weeks, or it fails. Most often the failure is a panic
> "malloc: out of space in kmem_map", but sometimes it just wedges. I've
> been operating on the assumption this is a kernel memory leak
> somewhere, most likely the timer socket code, but this makes it sound
> as though it might not be.
Unfortunately I never see such panics, just complete wedges. I think I
do remember seeing them on sparc and sun3 machines though, but I don't
push any of my remaining sparc machines very hard, and none have huge
amounts of memory, etc.
In my case I'm not sure it's a leak, but rather just a resource
starvation problem combined with some kind of deadlock. This machine is
a multi-CPU machine and I think something gets too far out of hand too
quickly such that by the time it's critical it's too late to back out of
what's currently being done to go release whatever resource is being
pushed over the limit.
I say this because on rare occasions I have seen a single process stuck
in "vmmapva" but it eventually unsticks. Some programs are more likely
to trigger the problem than others too it seems, such as gv+gs, or cvs,
though the most reliable trigger are the nightly /etc/{daily,security}
runs.
The sparc and i386 machines do seem to share the common problem that
they have very restricted KVA space. I had a hack in my 1.6 source tree
for sparc which I used to try to make sure nbufs weren't over-allocated.
On my SS20 with 320MB of RAM the 1.6 kernel says this on boot-up:
nbuf at 8170 is too large for VM_MAX_KERNEL_BUF... adjusted to 896
I've tried using a similar hack in the netbsd-4 tree however at the 1/4
limit it does not seem to trigger the restriction (or perhaps I'm doing
the calculation in two different units or I've made some other incorrect
assumption)
--- src/sys/arch/i386/i386/pmap.c 10 Feb 2007 13:03:29 -0500
1.194.2.1
+++ src/sys/arch/i386/i386/pmap.c 19 Jun 2009 12:24:21 -0400
@@ -73,6 +73,7 @@
#include <sys/systm.h>
#include <sys/proc.h>
#include <sys/malloc.h>
+#include <sys/buf.h>
#include <sys/pool.h>
#include <sys/user.h>
#include <sys/kernel.h>
@@ -818,7 +819,7 @@
vaddr_t kva_start;
{
struct pmap *kpm;
- vaddr_t kva;
+ vaddr_t kva, bufsz;
pt_entry_t *pte;
int i;
@@ -830,6 +831,20 @@
virtual_avail = kva_start; /* first free KVA */
virtual_end = VM_MAX_KERNEL_ADDRESS; /* last KVA */
+ /* Get the initial desired size of buffer cache */
+ bufsz = buf_memcalc();
+ /* if it seems too big then set an upper limit of 1/4 KVA */
+ if (bufsz > (virtual_end - virtual_avail) / 4) {
+ printf("WARNING: default bufmem too big: %ld;", bufsz);
+ bufsz = (virtual_end - virtual_avail) / 4;
+ printf("reduced to: %ld (set something lower!)\n", bufsz);
+ }
+ /*
+ * attempt to set a limit to the amount of memory that can be used for
+ * buffers, i.e. eventually known as bufmem_hiwater.
+ */
+ (void) buf_setvalimit(bufsz);
+
/*
* set up protection_codes: we need to be able to convert from
* a MI protection code (some combo of VM_PROT...) to something
--
Greg A. Woods
Planix, Inc.
<woods%planix.com@localhost> +1 416 218-0099 http://www.planix.com/
Attachment:
pgpGV1t2sX_be.pgp
Description: PGP signature