Re: percpu_foreach() does not execute remotely

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

> Date: Tue, 28 Jan 2020 20:49:50 -0800
> From: Jason Thorpe <thorpej%me.com@localhost>
> 
> What happens if:
> 
> 	oink = percpu_getref(...);
> 	...
> 	mutex_enter(...);	// blocks for a long time for whatever reason.
> 	// while we're blocked, someone else does a percpu_alloc() that results
> 	// in percpu_cpu_enlarge()?
> 	// Isn't "oink" invalid now?
> 	...
> 	percpu_putref(...);

Normally sleeping under percpu_getref -- even on an adaptive lock --
is a no-no for precisely this reason.  (percpu_getref, while very nice
in some ways, is an extremely delicate abstraction!  It would be nice
if violating this rule triggered an immediate panic.)

Do you mean to make it part of the contract of percpu_foreach_xcall
that sleeping is allowed in the callback, even though it would
generally be forbidden under percpu_getref?

I wasn't considering that case because I thought percpu_foreach_xcall
was intended for things like copying out statistics.

That said, if you do want to allow sleeping in the callback, I think
your reasoning is correct: during the percpu foreach xcall, no other
low-priority xcalls -- particularly percpu_cpu_swap -- can run; and
during percpu_cpu_swap, high-priority xcalls are blocked by splhigh
during the pcc->pcc_data = ... update.

If we commit to this semantics, I agree that it warrants a comment,
both in subr_percpu.c and in subr_xcall.c.