Some questions about x86/xen pmap

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

Hi lists,

I have some questions regarding the pmap of x86/xen, mainly for 
practical reasons; I am not very familiar with it.

Just to set the background, to make Xen save/restore/migrate work under 
a NetBSD domU, I have to do some slight modifications to the pmap, for 
two reasons:
- Xen (and xentools) do not handle alternative recursive mappings (the 
APDP entries in our PD).
- domains should not keep references to machine frame number (real 
physical page numbers) between save/restore, as they could change 
(especially when migrating VMs) without the domain noticing it.

I currently handle these issues that way:

- just before save, I am iterating through the pmap's to check that the 
PDIR_SLOT_APTE is set to 0. Otherwise, set it to 0.

- functions using MFNs are protected with rwlock(9) routines. Prior to 
using a MFN, the thread has to acquire a reader lock, and release it 
when it has finished with it. The suspension thread acquires a writer 
lock, so that every other thread that wants to manipulate MFNs have to 
wait for the suspension to release the writer, which is done during 
resume when it is deemed safe.

Questions are following:

- apart from the pmap_map_ptes() routines, which are used to map an 
arbitrary pmap inside the current one, is there another use of the APDP 
slot? I can't find any, but I may have missed it during my grep/read stuff.

- in pmap_unmap_ptes() (used to unlock the pmap used earlier by 
pmap_map_ptes()), the APDP is only unmapped when MULTIPROCESSOR is 
defined. While it is now the case for GENERIC, we do not have SMP 
support inside port-xen currently. Why is it necessary to empty the 
entry for MP and not for UP?

- what would be the fastest way to iterate through pmaps? My current 
code is inspired from the one found in pmap_extract(), but it requires 
to map/unmap each pmap to check for non-0 APDP slots. As they are not 
that common, this may be costly when the pmap list is lengthy. I tried 
to access the PD of each pmap through their pm_pdir member, but it 
usually ends badly with a fault exception in some cases.

- rwlock(9) clearly states that callers must not /recursively/ acquire 
read locks. Is a thread allowed to acquire a reader lock multiple times, 
provided it is not done recursively?

- the read/write locking is inspired from the one found in Solaris. If 
you have suggestion on doing the locking another way, feel free to 
suggest. I used rwlock(9) as it allows multiple readers, hence having 
minimal overhead in our case, while having write-preferring when domain 
is suspending.

Many thanks for your patience while reading, I tend to be a bit wordy in 
my mails.

Cheers!

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost