On Wed, Nov 28, 2007 at 07:06:43PM +0000, Andrew Doran wrote:
> On Wed, Nov 28, 2007 at 07:43:10PM +0100, Manuel Bouyer wrote:
> 
> > I see that the new atomic ops that have been added to kernel, are
> > patched to behave like non-atomic ops on UP kernels. 
> > 
> > Even for UP systems there are cases where we want atomic memory operations:
> > - read/write to in-memory structres shared with devices. Some device
> >   controllers really behaves like a CPU.
> 
> The atomic ops defined in sys/atomic.h are specified to work only with RAM,
> not device memory. Extending them to work reliably on device memory in a
> portable way would be .. an interesting challenge.

You mean, some systems can't do it.

> 
> > I know that reordered
> >   load/store can cause issues with esiop, and I suspect the occasional timeouts
> >   I've noticed can come from this. The lack of appropriate MI interface 
> >   to fix this has always bothered be, and I think it's a real issue.
> 
> Is this on x86?

Yes.

>  bus_space_barrier (a no-op) should be enough on x86 if the
> page is marked uncacheable, but I need to check that (i.e does PG_N imply
> strongly ordered access). bus_dmamap_sync needs an lfence, which I can see
> is missing. I will see about fixing that.

It looks strange to me to use bus_space() functions to order access to
DMA memory. If bus_dmamap_sync() does it even if there's no cache
flush or bounce buffers needed, at first glance it could be enough.

Does someone know if a device can do atomic memory operation though the
PCI bus ?

> 
> > - read-write to data shared with the hypervisor or another domain in Xen.
> >   A guest may be UP, although the hardware it's running on is SMP.
> >   Domain0 can eventually know the harware's number of CPUs but I'm
> >   not sure this information is available to guests (and with migration, this
> >   may change).
> 
> I feel this one is an x86/xen specific problem and should be handled there.
> I have just realised that the atomic ops will be patched on xen/amd64, will
> that cause problems for the pmap?

Right now it should not be patched AFAIK, because x86/x86/cpu is not
used. But I feel that one day we should call the patch() and errata()
functions, so we'll have to handle it.

> 
> On a tangent, there is some provision for dynamic reconfiguration with Xen,
> isn't there? That is: adding and removing virtual CPUs while the system is
> running.

Yes, it's theorically possible, but I don't know if it's functionnal
on linux right now.

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--