Re: Some questions about x86/xen pmap

[Andrew Doran <ad%netbsd.org@localhost>]

Hi,

Sorry for taking so long to reply.

On Mon, Apr 27, 2009 at 11:18:44PM +0200, Jean-Yves Migeon wrote:

> I have some questions regarding the pmap of x86/xen, mainly for 
> practical reasons; I am not very familiar with it.
> 
> Just to set the background, to make Xen save/restore/migrate work under 
> a NetBSD domU, I have to do some slight modifications to the pmap, for 
> two reasons:
> - Xen (and xentools) do not handle alternative recursive mappings (the 
> APDP entries in our PD).

I' planning to elminate these for native x86. Instead of using the alternate
space, native x86 will temporarily switch the pmap onto the CPU. I discussed
this briefly with Manuel and he suggested that there may be problems with
this approach for xen. My patch leaves it unchanged for xen.

> using a MFN, the thread has to acquire a reader lock, and release it 
> when it has finished with it. The suspension thread acquires a writer 
> lock, so that every other thread that wants to manipulate MFNs have to 
> wait for the suspension to release the writer, which is done during 
> resume when it is deemed safe.

I think I have seen a patch where you did this? There used to be a global RW
lock in the pmap and we worked quite hard to get rid of it. Global RW locks
are bad news. Even if there is no contention on the lock itself, they cause
lots of cache line ping-ponging between CPUs and lots of writebacks to main
memory. For heavyweight stuff that's OK but the pmap really gets hammered
on at runtime. As an alternative I would suggest locking all existing pmaps
and also prevent new pmaps from being created while your code runs. Would
that work for you?

> Questions are following:
> 
> - apart from the pmap_map_ptes() routines, which are used to map an 
> arbitrary pmap inside the current one, is there another use of the APDP 
> slot? I can't find any, but I may have missed it during my grep/read stuff.

Nope.
 
> - in pmap_unmap_ptes() (used to unlock the pmap used earlier by 
> pmap_map_ptes()), the APDP is only unmapped when MULTIPROCESSOR is 
> defined. While it is now the case for GENERIC, we do not have SMP 
> support inside port-xen currently. Why is it necessary to empty the 
> entry for MP and not for UP?

From memory this is a NULL assignment? I don't know.
 
> - what would be the fastest way to iterate through pmaps? My current 
> code is inspired from the one found in pmap_extract(), but it requires 
> to map/unmap each pmap to check for non-0 APDP slots. As they are not 
> that common, this may be costly when the pmap list is lengthy. I tried 
> to access the PD of each pmap through their pm_pdir member, but it 
> usually ends badly with a fault exception in some cases.

I believe that there is a global list of pmaps. Have a look at
pmap_growkernel().
 
> - rwlock(9) clearly states that callers must not /recursively/ acquire 
> read locks. Is a thread allowed to acquire a reader lock multiple times, 
> provided it is not done recursively?

In short: no. You can do it with rw_tryenter(), but note that rw_tryenter()
can't be called in a loop. Why: even if you hold the lock read held, it may
be "closed" for new read holds, because a writer is waiting on the lock and
the RW lock code has decided that the writer should get priority. It's to
prevent "starvation".