You'll have to excuse the dumb suggestion, but maybe dumb
	suggestions can start the seed of a good idea :)

On Tue, 6 Jun 1995, Chuck Cranor wrote:

> hi-
> 
>    i am looking for ideas on how to handle DVMA with VME devices under
> NetBSD/sparc.  here is the problem i am facing:
> 
>    the hardware maps the lower addresses of the VME bus into the kernel
> virtual address space.   VME address zero maps to DVMA_BASE (0xfff00000),
> and the mappings end at DVMA_END (0xfffc0000).   in order to do DMA
> to a VME device, the kernel's buffer must be mapped in that range
> of virtual addresses.   the address given to the hardware is simply
> the kernel virtual address minus DVMA_BASE.  for example, if there was 
> a kernel buffer at KVA 0xfffc8000 then the hardware would see it at
> VME address 0x8000.

	Sorry, is that 0xfff08000 rather than 0xfffc8000?
> 
>    in NetBSD/sparc, DVMA space is mapped by "phys_map" (a vm_map_t
> declared in vm/vm_kern.h).
> 
>    i have written a device driver for the xylogics 7053 VME/SMD disk
> controller ("xd").   all I/O requests come in via the xdstragegy() routine
> in "struct buf *"s.   there are two types of "struct buf *"s that my
> driver understands.    if (bp->b_flags & B_PHYS) != 0, then the buffer
> is from an I/O operation on the raw disk device (e.g. /dev/rxd0a) and
> the buffer is already mapped into DVMA space due to the vmapbuf()
> function in sparc/vm_machdep.c.    if (bp->b_flags & B_PHYS) == 0, then
> the buffer is part of the kernel's "struct buf" array, and is not
> mapped in DVMA space (but is in kernel space).   if that is the case, 
> then it must be mapped into DVMA space before I/O can begin.   also, 
> once I/O is done this mapping must be removed.
> 
>    what I wanted to do was to establish the mapping for the non-B_PHYS
> case before I started I/O and the remove it in the xdintr() routine.
> to test this, I ran a "while (1) ; fsck /dev/rxd0a ; end" in one window
> while doing "iozone" on /dev/xrd1g in another.    I discovered that my 
> scheme didn't work.   the problem was that the "fsck" would do an I/O 
> on the raw device and it would call vmapbuf().   this would cause phys_map 
> to be locked.  in the mean time an xd interrupt would happen (due to 
> the "iozone") and the interrupt routine would try and unmap a buffer 
> while the phys_map was locked by the vmapbuf() call.   this would cause 
> the interrupt to try and go to sleep in lock_write() by calling 
> thread_sleep() [vm/kern_lock.c].   this would cause my system to crash.
> 
>    i solved this problem by protecting all accesses to "phys_map"
> with splbio().   this works just fine.   however, it is unclear to
> me if phys_map can be handled this way.   in fact, pk and theo say 
> that it is unwise to call kmem_free_wakeup() at interrupt time and
> that I should try and to all my memory mappings in the context of a
> process rather than in an interrupt.   for the mapin, this is easy enough
> to do... you just map it in in the top of the strategy routine
> (although this is somewhat wasteful of DVMA space).   however, I am
> not sure what to do with the map out operation.   the map out is triggered
> by an interrupt.  one thing that was suggested was keeping a queue
> of buffers to map out in the device driver's softc structure and then wait
> until a non-interrupt time to do it (e.g. hope that someone calls xdstragegy
> soon?).   that seems pretty ugly to me.  what else can I do?   i'm
> looking for ideas or insight that can help me.
> 
	Please exuse my ignorance here, but I hope you'll allow me to throw
	out a few ideas & excuse the fact I don't know many basic premises.

	Presumably the kernel calls xdstragegy()  which sets everything 
	up & initiates the read request then exits, leaving the process in
	disk wait.
	Sometime later the interrupt of 'data ready' is triggered, which
	transfers the data across & marks the process as runnable.

	Is there a facility for xdstragegy() to mark the process in
	such a way the when it runs again it runs a routine in the
	kernel? xdbuffertidyup(), when then continues the process?

	Just a quick question: Are the buffers double mapped into kernel
	& DVMA space (I said I was ignorant)

	Then again, how big would your original ideas queue get?
	There shouldn't be that many processes waiting on xdstragegy()'s 
	interrupt result- and even if there were the odds are very high
	there would be more disk io coming from someone very soon after to
	clear the queue.

	Final dumb idea: Some DMA implementations on an ISA bus have 'bounce
	buffers' mapped in below 16Mb to permit DMA to those buffers, then
	the result is 'bounced' up above the 16Mb limit (& vica versa).
	You could have a pool of memory permenantly mapped into DVMA, and
	each interrupt routine would grab a block, do the DMA & then copy
	it across into the kernel buffer. The number of bounce buffers
	could be kept very low as there shouldn't be many simultaneous DVMA
	interrupts active...
	Its a permenant overhead of a few (one?) buffers, plus an extra
	memory to memory copy, but its simpler (if less efficient :( ).
	This feels off the mark, but I said it was dumb :)
	
		Thanks for your patience.

			David
  D.K.Brownlee@city.ac.uk (MIME) +44 171 477 8186 {post,host}master (abs) 
 Network Analyst, UCS, City University, Northampton Square, London EC1V 0HB.
   --- Bite and I'll bite back until one of us lies dead and bleeding ---