"Justin T. Gibbs" <gibbs@freefall.freebsd.org> writes:

> 
> >It appears all that was done was to make the start routines not use
> >SCSI_NOSLEEP when called from the strategy routine.  It is still set
> >when called from an interrupt (i.e. on command completion), so if you
> >have enough I/O pending to overflow the SCBs already allocated, you
> >will still lose.  This is made more noticable by ccd because it can
> >have the effect of queueing many requests at once to the same
> >controller.
> 
> So the difference becomes one of always setting SCSI_NOSLEEP or only
> setting it at interrupt time when it is valid.  At interrupt time, the only
> case (as I recall - haven't looked at the code lately) that we start a scsi
> command is for a retry of a failed command, which implies that the number
> of openings is >= 1.

That's distinctly untrue.  If there are no SCBs available when a
command is queued, and one can't be allocated without sleeping, then
the driver, under FreeBSD as well, will return TRY_AGAIN_LATER.  This
command will be retried for a few seconds, and if it can't be
executed, the SCSI layer will eventually give up.  This won't cause a
`not queued' message, but it will still do the wrong thing, and for
more or less the same reason.

> At least for a FreeBSD system running ccd to 6 disks
> of a 2940 (16 SCBs), the failure that was reported for NetBSD never occurs
> and I know that there are more than 16 active transaction in this system.

You could be getting lucky by allocating the SCBs earlier for some
other reason, but it's still broken.

> This is not a problem because only requests generated outside of an
> interrupt context will cause you to rise above your (previous) threashold.

That's not true, either.  If you increase `openings' in an interrupt
context (as is done in ahc_done()) and then wake up a higher-level
driver (through scsi_done()), the higher-level driver may immediately
attempt to queue more commands than there are currently SCBs available
for (from the interrupt context), and this lossage mode will ensue.

For this hack to work, `openings' must *only* be increased from
ahc_scsi_cmd(), when SCSI_NOSLEEP is not set.  Therefore there is a
bug in the ahc driver; it violates an invariant of the SCSI system.

> >As I've said before, the best solution is to eliminate the need for
> >SCSI_NOSLEEP.  This requires more restructuring of the SCSI code,
> >however.
> 
> Hey, that was what I said! B-) But you called my plan for doing it a hack.
> The plan was to basically restructure the SCSI system so that you only
> sleep in one place.  So, a process might sleep at the time it queued an I/O
> request waiting for its scsi_xfer struct to be allocated/reserved, but that
> once it got that structure, the controller resources would already be
> attached to it and no further sleeping would be needed.  This also ensures
> that all mallocing is done outside of an interrupt context.

That was *my* suggestion.  You proposed the previously mentioned hack
as an alternative.