Re: netbsd-6: pagedaemon freeze when low on memory

[David Young <dyoung%pobox.com@localhost>]

On Mon, Mar 04, 2013 at 02:43:47PM -0500, Richard Hansen wrote:
> With the patch applied, the pagedaemon no longer freezes.  However, LWPs
> start piling up in vmem_alloc() waiting for memory to become available.
>  So it seems like this change is necessary but not sufficient.

vmem_alloc(..., VM_SLEEP) will sleep forever while resources are not
available to fulfill the request, and I think that's good.  It looks to me
like vmem_alloc() needs to wake from its sleep and retry in either of
two conditions.

Condition 1: the backend has new memory available for vmem to add to
    the arena

    I don't think vmem_alloc() will ever wake in this condition,
    because I don't see any way for a backend to signal to the vmem
    that memory is available.

    To fix this problem, add a new vmem(9) method,
    vmem_backend_ready(vmem_t *, ...) and add a couple of vm_flag_t's,
    VM_SLEEPING and VM_WAKING.  Before vmem_alloc() starts to wait
    on its condition variable, let it call the backend's import
    callback with the VM_SLEEPING flag.  Now the backend knows the
    arena is sleeping and the parameters of the region it waits
    for.  If a region that may satisfy the VM_SLEEPING call is
    freed, let it call vmem_backend_ready() on the arena.  Let
    vmem_alloc() call the import function again with VM_WAKING when
    it has satisfied the request that it was VM_SLEEPING for.

Condition 2: the vmem arena was replenished with vmem_free() or vmem_xfree()

    vmem_alloc() sometimes will wake in this condition, however,
    I'm not sure that it will wake reliably, because it does not
    continuously hold a lock both while it checks the out-of-memory
    condition and while it waits on a condition variable for the
    out-of-memory condition to change.  There's a race condition:
    it's possible for a thread to alleviate the out-of-memory
    condition after vmem_alloc() tests for it but before vmem_alloc()
    does VMEM_LOCK(vm); VMEM_CONDVAR_WAIT(vm).  If that happens,
    then vmem_alloc() could wait forever for the out-of-memory
    condition to end.

    It's undesirable for vmem_alloc() to VMEM_LOCK(vm) unless it's
    strictly necessary, and testing the out-of-memory condition
    a second time after VMEM_LOCK() but before VMEM_CONDVAR_WAIT()
    seems redundant.

    Maybe we can avoid unnecessary locking or redundancy using a
    generation number?  Add a generation number to the vmem_t,
    
        volatile uint64_t vm_gen;

    Increase a vmem_t's generation number every
    time that vmem_free(), vmem_xfree(), or vmem_backend_ready() is
    called:

        VMEM_LOCK(vm);  /* have to hold lock to modify vm_gen */
        vm->vm_gen++;
        VMEM_CONDVAR_BROADCAST(vm);
        VMEM_UNLOCK(vm);

    Before testing the out-of-memory condition in vmem_alloc(),
    read the generation number:

    again:
        gen = vm->vm_gen;
        membar_consumer();

        ... memory available? if so, return.  otherwise ...

        VMEM_LOCK(vm);
        while (gen == vm->vm_gen)
                VMEM_CONDVAR_WAIT(vm);
        VMEM_UNLOCK(vm);
        goto again;

Dave

-- 
David Young
dyoung%pobox.com@localhost    Urbana, IL    (217) 721-9981