This is a little bit awkward to explain - so I better come up with
a little framework:

	MEM_PDE: A Page Directory Entry in Memory
	TLB_PDE: A Page Directory Entry cached in the TLB

	MEM_PTE: A Page Table Entry in Memory
	TLB_PTE: A Page Table Entry cached in the TLB
	
	PTP: Page containing MEM_PTEs

I am concerned about the following:

	1) A valid MEM_PDE points to a PTP containing only one valid MEM_PTE
	2) One or more processors cache the MEM_PDE ( TLB_PDE <= MEM_PDE)
	3) The last valid MEM_PTE in the PTP is zapped
	4) The MEM_PDE pointing to the PTP is zapped and the PTP is released to
	   the free page list.
           (user space pmap only)
	5) Several TLB entry flushes are queued.
           The entry queued to flush the TLB_PTE will also flush the TLB_PDE
	   ( I am not concerned about the entries that map the PTP into memory)
	6) Since the TLBs are not flushed yet the TLB_PDE cached in step 2 is still
	   pointing to the (now released) PTP  
	7) An interrupt or another processor takes the former PTP and fills it
	   with random data.
	8) The TLB_PDE cached in step 2 now points to a page containing random data.
	9) A "random table walk" has no reason to reload the cached TLB_PDE and as 
such
	   might load several random entries from the former PTP into the TLB.
           ( On multiprocessor systems another processor might actively try to 
access
            memory - no "random table walk" needed)  
	10) The entries loaded in step 9 (Lets call them TLB_PTE_INV_A,TLB_PTE_INV_B 
..)
	    could  have the global bit set and could  point to random valid or invalid
	    memory addresses.
	11) The TLB shootdown causes all the queued TLB flushes to be executed.
	    This causes the TLB_PDE and the TLB_PTE entries from step 1-3 to be
	    invalidated.
	12) Unfortunately TLB_PTE_INV_A,TLB_PTE_INV_B,.. from step 9-10 are still 
loaded
            in the TLB and might cause problems.


I have some additional concerns about reference bits and dirty bits and TLBs
as described in my email with subject "i386 TLB shootdown race condition ?".
However, it may better to postpone the discussion about these issues to avoid
too much confusion.

Stephan

---------------------
Stephan Uphoff 
ups@stups.com
9275 Martin Road
Roswell, GA 30076
770-518-4058
 


> [redirected to port-i386]
> 
> On Fri, Jan 31, 2003 at 11:02:50PM -0500, Stephan Uphoff wrote:
> > 
> > If a (user space) Page Table Page (PTP) is no longer used, because the last
> > valid Page Table Entry (PTE) has been removed, the associated page structure 
> > is put on the free page list before the associated TLB for the PTP 
> > (TLB that caches the PDE) is flushed.
> > 
> > Any modification of the freed page (interrupt, multiprocessor) might cause 
> > loading of invalid TLB entries that could:
> > 	- cause immediate problems for multiprocessor systems.
> > 	- stay in the TLB even after the call to pmap_tlb_shootnow().
> >           (and as such cause problems later).
> > 
> > This is not only a problem for multiprocessor systems as Intel states
> > in one of their "Pentium II Processor Application Notes":
> > 	Memory Ordering On Dynamic Execution (Pentium Pro Family) Processors
> >  	2.7. Page Table Walking Accesses:
> >   	     [...]
> >   	  *  Page table walks can occur at any time, randomly
> >    	     [...]
> > 		Page table walking to satisfy TLB (Translation Lookaside
> > 		Buffer) misses can be performed speculatively and out-of-order;
> >  		page table walks are subject to speculative cacheability.
> >   	     [...]
> 
> I don't quite see how a problem could occur here:
> 
> 	* When a PTP is freed, it means that it is no longer used
> 	  in the page table. When it is freed, the pmap is locked,
> 	  so other CPUs won't enter any mappings that would make the
> 	  page needed again.
> 	* uvm_pagefree returns the physical memory to the freelist,
> 	  which is fine; the physical memory can be re-used, as the
> 	  PDE entry referencing it has been zapped.
> 
> You seem to be saying that somehow a page table lookup can occur
> that would still reference the physical memory of the PTP. I
> don't see how that could happen, as virtual memory addresses that
> would cause the PTP to be referenced are all invalid. I also
> don't see how the CPU could ever set a referenced bit in a PTE
> entry in the PTP. "out of order" and "speculative", sure, but
> this would be more like "out of the blue and uncalled for"..
> I really don't think the CPU is going to do that.
> 
> What would be, in your opinion, a scenario which would cause
> a problem?
> 
> - Frank
> 
> -- 
> Frank van der Linden                                    fvdl@wasabisystems.com
> ==============================================================================
> Quality NetBSD Development, Support & Service.   http://www.wasabisystems.com/
>