netbsd-6: pagedaemon freeze when low on memory

[Richard Hansen <rhansen%bbn.com@localhost>]

Hi all,

I believe I have found a bug in the pagedaemon (uvm_pageout() in
src/sys/uvm/uvm_pdaemon.c) that causes the system to freeze when the
kmem_arena runs low (<10% free):

  1. line 254: uvm_km_va_starved_p() returns true
  2. line 258: the !kmem_va_starved condition prevents the pagedaemon
from sleeping
  3. lines 330--346: no memory is freed -- it's all still in use
  4. go to step #1

To reproduce the freeze:

  1. acquire an i386 system with 4GB of memory and lots of files in the
filesystem
  2. set the kern.maxvnodes sysctl as high as it will go
  3. run 'du -skx /'
  4. wait for the kernel to run low on memory (vnode allocations)

I believe the bug was introduced in this commit:
  http://mail-index.netbsd.org/source-changes/2012/02/01/msg031411.html
  http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/uvm/uvm_pdaemon.c#rev1.105

Attached is an initial attempt at fixing this.  The patch allows the
pagedaemon to sleep if no memory was reclaimed the last time through the
loop.  This is probably not a correct or complete fix; I don't yet have
a comfortable understanding of the inner workings of the kernel.

With the patch applied, the pagedaemon no longer freezes.  However, LWPs
start piling up in vmem_alloc() waiting for memory to become available.
 So it seems like this change is necessary but not sufficient.

Thoughts?

Thanks,
Richard

allow pgdaemon to sleep if previous pass didn't free anything

Before, the pagedaemon would keep tring over and over to drain pools
and buffer caches whenever the amount of free space in the
`kmem_arena` dropped below 10% ("starved").  This caused the system to
freeze if draining didn't actually free any memory.

Now the pagedaemon will sleep when starved if the previous pass didn't
free any memory.
---
 src/sys/uvm/uvm_pdaemon.c |   24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/src/sys/uvm/uvm_pdaemon.c b/src/sys/uvm/uvm_pdaemon.c
index f50d638..9af6f52 100644
--- a/src/sys/uvm/uvm_pdaemon.c
+++ b/src/sys/uvm/uvm_pdaemon.c
@@ -229,6 +229,19 @@ uvm_pageout(void *arg)
        int extrapages = 0;
        struct pool *pp;
        uint64_t where;
+       /*
+        * freed_something is true if the previous pass in the main
+        * loop managed to reclaim some memory, false otherwise.  This
+        * is used to prevent the system from freezing when the system
+        * is "starved" and no memory can be reclaimed (starved ->
+        * unable to reclaim anything -> still starved -> still unable
+        * to reclaim anything -> ...).
+        *
+        * initialized to true so that the pagedaemon isn't forced to
+        * sleep at the beginning of the first pass if starved
+        * (unlikely to happen, but just in case...)
+        */
+       bool freed_something = true;
        
        UVMHIST_FUNC("uvm_pageout"); UVMHIST_CALLED(pdhist);
 
@@ -250,12 +263,14 @@ uvm_pageout(void *arg)
 
        for (;;) {
                bool needsscan, needsfree, kmem_va_starved;
+               bool buf_drained_something;
+               bool pool_drained_something;
 
                kmem_va_starved = uvm_km_va_starved_p();
 
                mutex_spin_enter(&uvm_fpageqlock);
                if ((uvm_pagedaemon_waiters == 0 || uvmexp.paging > 0) &&
-                   !kmem_va_starved) {
+                   (!kmem_va_starved || !freed_something)) {
                        UVMHIST_LOG(pdhist,"  <<SLEEPING>>",0,0,0,0);
                        UVM_UNLOCK_AND_WAIT(&uvm.pagedaemon,
                            &uvm_fpageqlock, false, "pgdaemon", 0);
@@ -337,13 +352,16 @@ uvm_pageout(void *arg)
                 * kill unused metadata buffers.
                 */
                mutex_enter(&bufcache_lock);
-               buf_drain(bufcnt << PAGE_SHIFT);
+               buf_drained_something = buf_drain(bufcnt << PAGE_SHIFT);
                mutex_exit(&bufcache_lock);
 
                /*
                 * complete draining the pools.
                 */
-               pool_drain_end(pp, where);
+               pool_drained_something = pool_drain_end(pp, where);
+
+               freed_something =
+                   buf_drained_something || pool_drained_something;
        }
        /*NOTREACHED*/
 }
-- 
1.7.9.3