On Wed, 9 Nov 2011 08:57:08 -0500, Thor Lancelot Simon wrote:
On Wed, Nov 09, 2011 at 11:49:10AM +0100, Jean-Yves Migeon wrote:This is the idea. One shortcoming though: the pool_cache(9) content is not synchronously invalidated, depleting the per-CPU cache would only happen when the CPU pool_cache_get() a new object. From an API perspective is it ok for everyone? In my case, I'll have to xcall(9) a pool_cache_get() for each CPU to force-deplete the pools because of Xen shortcomings (it tracks page types and will raise an exception when pages are not freed on a suspension).I guess I don't quite understand what's happening on suspension. You are talking about suspending the Xen VM? Or taking a CPU offline?
When a page is allocated from the pmap_pdp_cache [1] it gets constructed and pinned as a L2 page. The type associated to a page is only destructed in dtor.
Upon suspend, Xen will save all states for pages. When resuming, the hypervisor will deduce the types given the content of the pages. Unfortunately, it handles our recursive mappings poorly and tries to pin as L1 a page pinned as L2. This leads to instant-kill of the domU.
To avoid this, I remove all recursive mappings from pmaps on suspend and reinstate them on resume. However pmaps "cached" in the pool_cache(9) are only known to the pool internals, so I can't fix them; only way is to invalidate/drain the pool to force dtor calls. hence, pool_cache_invalidate().
If I use a per-CPU serial flag, the pool will only get drained upon next pool_cache_get(). So I have to xcall(9) a _get() to effectively destroy the per-CPU cached items.
[1] http://nxr.netbsd.org/xref/src/sys/arch/x86/x86/pmap.c#2014 -- Jean-Yves Migeon jeanyves.migeon%free.fr@localhost