Thanks for your long response. I'll just respond to the last paragraph.

> In the case of sparc64 (which is where all the problems are, apparently),
> the COHERENT bit could be used to hint that the CPU mapping of the memory
> should be un-cached.  And when the map is loaded with the CPU-mapped buffer,
> the IOMMU PTE bits can get the "coherent" information.  But, even if the
> IOMMU didn't get the "coherent" information, bus_dmamap_sync() operations
> for the ISP mailbox, which MUST happen around CPU access to the mailbox,
> must flush any I/O caches that may be managed by the IOMMU.

"syncing" doesn't work here. It's mapping that does the right thing. And
bounce buffers for a major platorm isn't the right answer. The assumption
that a post-mapping 'sync' operation exists is a flawed assumption.

What I'm going to do is to parse your mail again and update the man page with
mention of the ordering dependency and bus_dmamap_load_raw usages- after all,
without proper documentation things will be done in the 'wrong' order. Note
that doing things in the 'right' order wouldn't fix the problems on sparc64.

Now, you say that the correct order is:

        bus_dmamap_create(...);

        bus_dmamem_alloc(...);

        bus_dmamem_map(...);

        bus_dmamap_load(...);


So, the assumption here is that you have to have CPU
(BUS_DMA_COHERENT) mapping in order to infer you want
an byte-coherent IOMMU mapping. This doesn't handle the
case of two non-cpu devices sharing memory, but that is,
I would agree, a bit of an edge case.

So.. follow the logic here....

The inference of all of this is that

1. Prior to a bus_dmamap_load, a bus_dmamem_map has to be done to give the
'hint' that BUS_DMA_COHERENT is to be used.

2. But real memory is allocated (in the order above) prior to the call to
bus_dmamem_map.


3. Therefore, bus_dmamem_alloc'd memory must be assumed to be for
BUS_DMA_COHERENT purposes because there may be no way to 'change'
the identity of the memory alloc'd in bus_dmamem_alloc (it might be
from a pool that *can't* be made byte-coherent).


So the required order implies that bus_dmamem_alloc memory must be capable of
being made BUS_DMA_COHERENT (if bus_dmamem_map'd) *and* IOMMU coherent (if you
want to distinguish this from BUS_DMA_COHERENT) in case you *don't* map into
the CPU's virtual address space.

Which means that of the two choices I summarized and presented, you're arguing
for this rather than allowing usage hints via functions other than
bus_dmamem_map.

As I said- *I* don't care which one of either. I just want it to be
clearer. Hopefully Eduardo can weigh in and we can come to closure on this
and move on.

-matt