Re: Problem with outstanding knotes and device detach - and a fix

[Jason Thorpe <thorpej%me.com@localhost>]

> On Jul 12, 2022, at 7:54 AM, Jason Thorpe <thorpej%me.com@localhost> wrote:
> 
> The following patch rectifies this situation by having klist_fini() traverse a list of knotes and substitute their filterops with no-op stubs.  This requires synchronizing with any calls into the filterops themselves.  We have convenient funnel-points for this in kern_event.c already, so it’s not particularly difficult… the main issue is that the filterops themselves are allowed to block.  My solution to this was to have a single global rwlock, knote_filterops_lock, that’s acquired for reading when a call though a knote's filterops is made, and in klist_fini() is acquired for writing for the short duration needed to stub out filterops.  This lock should **very rarely** be contended, but it will be *hot* (lots of acquire-for-read), so it gets its own cache line.
> 
> If someone has ideas about another synchronization mechanism, I’m all ears… again, the main issue is that the filterops calls themselves are allowed to block, so I’m not sure that any of our passive serialization mechanisms would work here.

FWIW, another alternative is to put the rwlock into the knote itself, if we think that lock will be Too Hot (which is not an unreasonable concern on a machine with many cores).  The trade-off is memory.  I could see putting the lock in the knote itself on amd64 and aarch64 (where we would expect to see high core count machines) and a global lock on everything else.  Then again, on amd64, struct knote is 136 bytes, so it already spills into the kmem-00192 bucket, and on i386 it’s 84 bytes, so kmem-00128 bucket.  I guess each of those could easily accommodate an additional pointer-sized struct member to alleviate any scalability concern about a global rwlock.

-- thorpej