Re: kern/53124 (FFS is slow because pmap_update doesn't scale)

[Maxime Villard <max%m00nbsd.net@localhost>]

Le 24/03/2018 à 23:07, Michael van Elst a écrit :
> On Sat, Mar 24, 2018 at 07:10:03PM +0000, maxv%NetBSD.org@localhost wrote:
>
> > My guess is that the 'pmap_update' you're talking about actually
> > touches a user pmap, and not pmap_kernel.
>
> ubc_alloc touches pmap_kernel when deleting old mappings in the UBC
> window. It is this call that is slow:
>
> 	/*
> 	 * Mapping must be removed before the list entry,
> 	 * since there is a race with ubc_purge().
> 	 */
> 	if (umap->flags & UMAP_MAPPING_CACHED) {
> 			umap->flags &= ~UMAP_MAPPING_CACHED;
> 			mutex_enter(oobj->vmobjlock);
> 			pmap_remove(pmap_kernel(), va,
> 				va + ubc_winsize);
> 			pmap_update(pmap_kernel());
> 			mutex_exit(oobj->vmobjlock);
> 	}

Alright, so my initial guess was wrong.

> [...]
> > But this guess would have to be verified. You should probably try to
> > assign your program to a given core - and this, early, _before_ your
> > program starts doing heavy stuff. schedctl, or pset would be even
> > better. If I'm right, it should "fix" the slowdown.
>
> Why would the synchronization with other CPUs go away?

Because if my initial guess was correct, by assigning your program to a cpu
you would have guaranteed that your pmap was not loaded on other cpus; and
as a result you wouldn't have had to send IPIs to them when changing a page
in your pmap.

But I may have another guess now. Here the call path is: pmap_remove ->
pmap_remove_ptes -> pmap_remove_pte -> pmap_tlb_shootdown. You can see that
the last one gets called only when ((opte & PG_U) != 0). That is, only when
the page has already been used (read/write) by a core on the machine.

It would be nice if you could verify, somehow, two things:

 o Does the 'UMAP_MAPPING_CACHED' branch get taken the same way with and
   without your N-1 user threads. If there is a clear difference here, then
   it means the problem is that UBC does not scale. Otherwise:

 o When the 'UMAP_MAPPING_CACHED' branch gets taken, does the 'PG_U' one get
   taken too afterwards with and without your N-1 user threads. If there is a
   clear difference here, then it means that for some reason an idle kernel
   thread on a remote cpu decided to touch the page before you did yourself,
   which resulted in PG_U being set, and the IPIs being sent to everybody.

Maxime