Subject: Re: vnode usage and reclaimation - feels like deadlocking
To: Stephen M. Jones <smj@cirr.com>
From: Stephen M. Jones <smj@cirr.com>
List: tech-net
Date: 01/16/2004 18:14:29
I've experienced vnlock deadlockish behaviour twice today since increase
kern.maxvnodes to ~25% (250000) of system memory (1GB). Both clients
locked up just about the same time, although only one had few complaints
about the fileserver not responding. The interesting thing is that
one had 77122 vnodes used while the other had about 65000 .. still, there
was much delay from vnlocks, so much that the both clients had to be
dropped to the debugger which showed a majority of the processes (under
300 processes total) in a 'vnlock' state. This particular lock is
initialised on line 537 or vfs_subr.c:
vp->v_type = VNON;
vp->v_vnlock = &vp->v_lock;
lockinit(vp->v_vnlock, PVFS, "vnlock", 0, 0);
cache_purge(vp);
vp->v_tag = tag;
vp->v_op = vops;
insmntque(vp, mp);
*vpp = vp;
vp->v_usecount = 1;
vp->v_data = 0;
simple_lock_init(&vp->v_uobj.vmobjlock);
I was able to get crash dumps of both clients and have since rebooted
them. (anyone have a software watchdog that would crash dump a system
when it hangs like this?)
NFS client one's usage:
Fri Jan 16 22:46:33 UTC 2004 6024 active vnodes
Sat Jan 17 00:06:37 UTC 2004 22168 active vnodes
NFS client two's usage:
Fri Jan 16 22:46:31 UTC 2004 7837 active vnodes
Sat Jan 17 00:06:35 UTC 2004 32560 active vnodes
The fileserver, which before only had limits of 32k and shortly 64k for
kern.maxvnodes is now using 166843 vnodes.
The other NFS clients seem to be doing fine so far with their usage as:
Three: Sat Jan 17 00:07:05 UTC 2004 70617 active vnodes
Four: Sat Jan 17 00:07:26 UTC 2004 65451 active vnodes
Five: Sat Jan 17 00:07:41 UTC 2004 98114 active vnodes