>
>"Justin T. Gibbs" <gibbs@freefall.freebsd.org> writes:
>
>> 
>> >> The implication is that your SCSI controller is failing to send the 
>> >> command.  It could also be that the SCSI subsystem is unable to malloc() 
>> >> storage for a data structure (note the SCSI_NOSLEEP).
>> 
>> Ahhh.  The infamous SCSI_NOSLEEP problem.  This was fixed in FreeBSD more
>> than a year ago (when wcarchive was using a 2742 controller) and is
>> caused by the SCSI system setting SCSI_NOSLEEP more often then required.
>
>It appears all that was done was to make the start routines not use
>SCSI_NOSLEEP when called from the strategy routine.  It is still set
>when called from an interrupt (i.e. on command completion), so if you
>have enough I/O pending to overflow the SCBs already allocated, you
>will still lose.  This is made more noticable by ccd because it can
>have the effect of queueing many requests at once to the same
>controller.

So the difference becomes one of always setting SCSI_NOSLEEP or only
setting it at interrupt time when it is valid.  At interrupt time, the only
case (as I recall - haven't looked at the code lately) that we start a scsi
command is for a retry of a failed command, which implies that the number
of openings is >= 1.  At least for a FreeBSD system running ccd to 6 disks
of a 2940 (16 SCBs), the failure that was reported for NetBSD never occurs
and I know that there are more than 16 active transaction in this system.
When CCD queues its requests, we are not in an interrupt context, and its
perfectly valid for the device driver to sleep.

>As we discussed at Usenix, one way to hack around this is to increase
>the `openings' (correct English spelling, BTW B-))

I believe this depends on your continent of reference.  Julian would
have to say for sure why he spelled it that way.

>count from
>*_scsi_cmd(), only when a command is queued with SCSI_NOSLEEP cleared,
>and to ensure when it is changed that all of the data structures
>needed for all openings on all devices attached to that controller are
>preallocated.  The ahc driver does not do this.

This is not a problem because only requests generated outside of an
interrupt context will cause you to rise above your (previous) threashold.
The NetBSD drivers must also be mallocating when SCSI_NOSLEEP is set (as
will the FreeBSD drivers although as I've said above, this situation never
arises) since this flag is always set in NetBSD.  There is currently no
protection against mallocating at interrupt time. (Can you malloc(NO_WAIT)
in an interrupt context safely?)  Regardless, this doesn't solve the
problem of needing to sleep when all resources are currently in use and
more cannot be created.  This also wasn't quite what we were talking about
at USENIX.

>As I've said before, the best solution is to eliminate the need for
>SCSI_NOSLEEP.  This requires more restructuring of the SCSI code,
>however.

Hey, that was what I said! B-) But you called my plan for doing it a hack.
The plan was to basically restructure the SCSI system so that you only
sleep in one place.  So, a process might sleep at the time it queued an I/O
request waiting for its scsi_xfer struct to be allocated/reserved, but that
once it got that structure, the controller resources would already be
attached to it and no further sleeping would be needed.  This also ensures
that all mallocing is done outside of an interrupt context.

--
Justin T. Gibbs
===========================================
  FreeBSD: Turning PCs into workstations
===========================================