Re: Am I using bus_dma right?

[Eduardo Horvath <eeh%NetBSD.org@localhost>]

On Thu, 23 Apr 2020, Mouse wrote:

> Okay, here's the first problem.  There is no clear "transaction
> completes".

Let's clarify that.

> The card has a DMA engine on it (a PLX9080, on the off chance you've
> run into it before) that can DMA into chained buffers.  I set it up
> with a ring of butters - a chain of buffers with the last buffer
> pointing to the first, none of them with the "end of chain" bit set -
> and tell it to go.  I request an interrupt at completion of each
> buffer, so I have a buffer-granularity idea of where it's at, modulo
> interrupt servicing latency.
> 
> This means that there is no clear "this transfer has completed" moment.
> What I want to do is inspect the DMA buffer to see how far it's been
> overwritten, since there is a data value I know cannot be generated by
> the hardware that's feeding samples to the card (over half the data
> pins are hardwired to known logic levels).
> 
> I've been treating it as though my inspection of a given sample in the
> buffer counts as "transfer completed" for purposes of that sample.

Are you inspecting the buffer only after reciept of an interrupt or are 
you polling?  

> 
> > When you do a write operation you should:
> 
> > 1) Make sure the buffer contains all the data you want to transmit.
> 
> > 2) Do a BUS_DMASYNC_PREWRITE to make sure any data that may remain in
> > the CPU writeback cache is flushed to memory.
> 
> > 3) Tell the hardware to do the write operation.
> 
> > 4) When the write operation completes... well it shouldn't matter.
> 
> ...but, according to the 8.0 manpage, I should do a POSTWRITE anyway,
> and going under the hood (this is all on amd64), I find that PREREAD is
> a no-op and POSTWRITE might matter because it issues an mfence to avoid
> memory access reordering issues.

I doubt the mfence does much of anything in this circumstance, but 
POSTWRITE does tell the kernel it can free up any bounce buffers it 
may have allocated if it allocated bounce buffers, but I digress.

> 
> > If you have a ring buffer you should try to map it CONSISTENT which
> > will disable all caching of that memory.
> 
> CONSISTENT?  I don't find that anywhere; do you mean COHERENT?

Yes COHERENT.  (That's what I get for relying om my memory.)

> 
> > However, some CPUs will not allow you to disable caching, so you
> > should put in the appropriate bus_dmamap_sync() operations so the
> > code will not break on those machines.
> 
> For my immediate needs, I don't care about anything other than amd64.
> But I'd prefer to understand the paradigm properly for the benefit of
> potential future work.

I believe if you use COHERENT on amd64 none of this matters since it turns 
off caching on those memory regions.  (But I don't have time to grep the 
souces to verify this.)


> > Then copy the data out of the ring buffer and do another
> > BUS_DMASYNC_PREREAD or BUS_DMASYNC_PREWRITE as appropriate.
> 
> Then I think I was already doing everything necessary.  And, indeed, I
> tried making the read routine do POSTREAD|POSTWRITE before and
> PREREAD|PREWRITE after its read-test-write of the samples, and it
> didn't help.

Ah now we're getting to something interesting.

What failure mode are you seeing?

> >> One of the things that confuses me is that I have no write-direction
> >> DMA going on at all; all the DMA is in the read direction.  But
> >> there is a driver write to the buffer that is, to put it loosely,
> >> half of a write DMA operation (the "host writes the buffer" half).
> > When the CPU updates the contents of the ring buffer it *is* a DMA
> > write,
> 
> Well, maybe from bus_dma's point of view, but I would not say there is
> write-direction DMA happening unless something DMAs data out of memory.
>
> > even if the device never tries to read the contents, since the update
> > must be flushed from the cache to DRAM or you may end up reading
> > stale data later.
> 
> So I have to treat it like a DMA write even if there is never any
> write-direction DMA actually going on?

Yes.

> Then the problem *probably* is not bus_dma botchery.

Eduardo