Re: Why NetBSD x86's bus_space_barrier does not use [sml]fence?

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

> Date: Sat, 30 Nov 2019 19:47:17 +0900
> From: Shoichi Yamaguchi <s.ymgch228%gmail.com@localhost>
> 
> > Are you using BUS_SPACE_MAP_PREFETCHABLE?
> 
> No, the driver does not use it.

And it doesn't use mtrrs or BUS_SPACE_MAP_CACHEABLE either, right?

Just to be sure, can you run
<https://www.NetBSD.org/~jmcneill/old/mtrr.c> to confirm there are no
mtrrs on the system affecting the bus space region in question?

> > It no, the CALL instruction that calls bus_space_barrier() produces a write
> > to the stack when storing the return address.  On x86, stores are totally
> > ordered, and loads are never reordered around stores.  No further barrier
> > is needed.  This is the idea anyway, sometimes reality does not match..
> 
> Should NetBSD need to use the fences to *completely* prevent
> a reorder as other BSDs although the CALL instruction can
> typically avoid a reorder?

It would be better to understand _why_ the change has that effect.

Normally SFENCE is needed only for non-temporal stores (MOVNT) and
ordering CLFLUSH-issued stores, as far as I know, neither of which
should appear in MI drivers.

For uncacheable memory regions (which is what you get if you don't
specify BUS_SPACE_MAP_PREFETCHABLE or BUS_SPACE_MAP_CACHEABLE), loads
and stores are never reordered.  So there should be no need for LFENCE
or MFENCE on such regions either.

(It's not clear to me why we bus_space_barrier doesn't need LFENCE or
MFENCE with BUS_SPACE_MAP_PREFETCHABLE or BUS_SPACE_MAP_CACHEABLE, but
if you're not using those here, that shouldn't matter.)

One possible explanation is a missing bus_dmamap_sync -- that _may_
need LFENCE/MFENCE, and changing bus_space_barrier might have the side
effect of doing the job that bus_dmamap_sync should have done.

Can you share the driver source?