Vnode life cycle: add vnode state

[J. Hannken-Illjes <hannken%eis.cs.tu-bs.de@localhost>]

1) Objects vnode and vcache_node always exist together so it makes sense
to merge them into one object.  This can be done in two ways:

- Add the current vcache_node elements (hash list, key and flag) to the
  vnode.
- Overlay the vnode into vcache_node.  This way we also get two views
  to vnodes, a public one (struct vnode) visible from file system
  implementations and a private one (struct vcache_node) visible from
  the VFS subsystem (sys/kern/vfs_*) only.

I prefer the second way, a diff is here:
  http://www.netbsd.org/~hannken/vstate/001_merge_vnode_and_vcache_node


2) The state of a vnode is currently determined by its flags
VI_MARKER, VI_CHANGING, VI_XLOCK and VI_CLEAN.  These flags are not
sufficient to describe the state a vnode is in.  It is for example
impossible to determine if a vnode is currently attaching to a file system.

I propose to replace these flags with a vnode state, one of:

- VN_MARKER: Used as a marker to traverse lists of vnodes, will never change
  or appear in a regular vnode operation.
- VN_LOADING: Vnode is currently attaching to its file system and loading
  or creatingthe inode.
- VN_ACTIVE: Vnode is fully initialised and useable.
- VN_BLOCKED: Vnode is active but cannot get new references through vget().
- VN_RECLAIMING: Vnode is in process to detach from file system.
- VN_RECLAIMED: Vnode is (no longer) attached to its file system, its dead.

and these operations to work on the vnode state:

- VSTATE_GET(vp): Get the current state.
- VSTATE_CHANGE(vp, from, to): Change state.
- VSTATE_WAIT_STABLE(vp): Wait until the vnode is in stable state, one of
  VN_ACTIVE or VN_RECLAIMED.
- VSTATE_ASSERT(vp, state): Assert the current state.

All operations are protected with vp->v_interlock with one exception: to
change state from VN_LOADING vcache.lock is also needed.  We can't take
vp->v_interlock while the vnode is loading as it may change while
the file system initialises the node, see uvm_obj_setlock().

Possible state transitions are:

- <undefined> -> VN_LOADING: Node gets allocated in vcache_alloc().
- VN_LOADING -> VN_ACTIVE: Node has been initialised in vcache_get() or
  vcache_new() and is ready to use.
- VN_ACTIVE -> VN_RECLAIMING: Node starts disassociation from underlying
  file system in vclean().
- VN_RECLAIMING -> VN_RECLAIMED: Node finished disassociation from
  underlying file system in vclean().
- VN_ACTIVE -> VN_BLOCKED: Either vcache_rekey*() is changing the vnode key
  or vrelel() is about to call VOP_INACTIVE().
- VN_BLOCKED -> VN_ACTIVE: The blocking condition is over.
- VN_LOADING -> VN_RECLAIMED: Either vcache_get() or vcache_new() failed to
  associate the underlying file system or vcache_rekey*() drops a vnode
  used as placeholder.

A diff adding the state operations is here:
  http://www.netbsd.org/~hannken/vstate/002_vnode_state_operations

A diff implementing the state with these operations is here:
  http://www.netbsd.org/~hannken/vstate/003_use_vnode_state


Comments or objections anyone?

--
J. Hannken-Illjes - hannken%eis.cs.tu-bs.de@localhost - TU Braunschweig (Germany)