Subject: Re: vnode usage and reclaimation - feels like deadlocking
To: Stephen M. Jones <smj@cirr.com>
From: Stephen M. Jones <smj@cirr.com>
List: tech-net
Date: 01/16/2004 18:14:29
I've experienced vnlock deadlockish behaviour twice today since increase
kern.maxvnodes to ~25% (250000) of system memory (1GB).  Both clients
locked up just about the same time, although only one had few complaints 
about the fileserver not responding.  The interesting thing is that 
one had 77122 vnodes used while the other had about 65000 .. still, there
was much delay from vnlocks, so much that the both clients had to be
dropped to the debugger which showed a majority of the processes (under
300 processes total) in a 'vnlock' state.  This particular lock is
initialised on line 537 or vfs_subr.c:

        vp->v_type = VNON;
        vp->v_vnlock = &vp->v_lock;
        lockinit(vp->v_vnlock, PVFS, "vnlock", 0, 0);
        cache_purge(vp); 
        vp->v_tag = tag;
        vp->v_op = vops; 
        insmntque(vp, mp); 
        *vpp = vp;
        vp->v_usecount = 1;
        vp->v_data = 0;
        simple_lock_init(&vp->v_uobj.vmobjlock);

I was able to get crash dumps of both clients and have since rebooted
them.  (anyone have a software watchdog that would crash dump a system
when it hangs like this?)

NFS client one's usage:
Fri Jan 16 22:46:33 UTC 2004  6024 active vnodes
Sat Jan 17 00:06:37 UTC 2004  22168 active vnodes

NFS client two's usage:
Fri Jan 16 22:46:31 UTC 2004  7837 active vnodes
Sat Jan 17 00:06:35 UTC 2004  32560 active vnodes

The fileserver, which before only had limits of 32k and shortly 64k for
kern.maxvnodes is now using 166843 vnodes.

The other NFS clients seem to be doing fine so far with their usage as:
Three: Sat Jan 17 00:07:05 UTC 2004  70617 active vnodes
Four:  Sat Jan 17 00:07:26 UTC 2004  65451 active vnodes
Five:  Sat Jan 17 00:07:41 UTC 2004  98114 active vnodes