> I assume there is an inter cpu message to cause the flush to happen,
> so cpu's which don't have a cache entry, still get a chance to
> decrement the ref count.

So, I had a quick look... 

If smp_cache_flush() were to initialize a semaphore before
entering the for() loop, and each cpu did a V(sem) when they
had flushed, then at the end of the loop, do another for() loop doing
P(sem), and it would wait for the last cpu to flush.

Of course you'd need a semaphore for each cache line to avoid the
above creating a big bottleneck for cache flushing.  And of course the
semaphores would need to be in uncached memory.

And can you have two cpu's wanting to initiate a flush of the same
line(s) at the same time?

I expect there's a much better alternative... 

--sjg