Re: libc.so's vmobjlock / v_interlock

[Andrew Doran <ad%netbsd.org@localhost>]

On Sun, Jan 19, 2020 at 10:08:12PM +0000, Taylor R Campbell wrote:

> > Date: Sun, 19 Jan 2020 21:50:24 +0000
> > From: Andrew Doran <ad%netbsd.org@localhost>
> > 
> > The biggest single remaining concurency problem is the mutex on libc.so's
> > uvm_object / vnode.  It pops up in a number of places, but most clearly seen
> > above the "uvm_fault_internal" box.
> > 
> > I think making the vmobjlock / v_interlock a rwlock could help with making
> > inroads on this problem, because in the libc.so case it's not modifying too
> > much under cover of the lock (mostly pmap state).
> > 
> > [...]
> > 
> > Thoughts?
> 
> I think we should try to go straight to pserialized page lookup and
> skip rwlocks.  Should be relatively easy to teach radixtree.c to work
> pserialized.
> 
> We probably won't want to use pserialize_perform() but can perhaps
> instead use an epoch-based system with page daemon queues of dying
> pages that may still be referenced under pserialize_read.

This is an intriguing idea and I suspect that you've already thought about
it quite a bit but I don't think we're there just yet because there's more
to be dealt with first, namely that vmobjlock/v_interlock covers a lot more
than the index of pages in the object.

For example right now it's covering the usecounts, the flags in the vnode
and the page, the PV entries in the pmap, interlocking the long term page
locks (PG_BUSY), correct serialization of page visibility and TLB shootdown,
and a whole lot more.

I'm not fond of RW locks as a synchronisation primitive, there are usually
better ways to do things, but I like the RW lock idea in this instance for
two reasons.

Firstly RW locks are pretty easy to understand and use so we can hopefully
get a nice win for the system without it being too taxing in terms of the
amount of work needed or the potential for introducing bugs.  Secondly I
think it would enable us incrementally attack some of the areas where we
have what's really per-page state being covered by a per-object lock, which
I think ties in with what you're suggesting.

Andrew