In message <1116360410.7597.680.camel@palm>,
Stephan Uphoff writes:

>While there are no IPLs in FreeBSD any more - part of the concept kind
>of survived the transformation.
>The interrupt handlers (with exceptions) normally do not call any device
>driver functions. Instead they schedules an interrupt thread that is
>responsible for calling the device driver's interrupt function.
>The priorities of the interrupt threads are loosely based on previous
>spl levels allowing interrupt threads with better priority to interrupt
>(preempt) threads with lower priority.

But of course.  I've had a few private discussions with Robert Watson
and Sam Leffler about this issue, and about what FreeBSD-5 has done.
If I could've made it to BSDCan, I would gladly have had more.


OSF/1 aka Ditigal Unix aka Tru64 did something similar to the above;
if (dim) memory serves, they called it part of their two-level
scheduler.  However, that approach has one *huge* drawback: on any
architecture that doesn't have an address-space ID, the context-switch
costs are prohibitive.  (I'm simplifying, but you get the point).

I've also heard the GSN (HIPPI on steroids) throughput numbers on IRIX
relied heavily on high-context switch and interrupt-service rates,
from a combination of good memory bandwidth, good caches, and ASIDs
(both from IRIX engineers and former colleagues working on what was
then SimOS/DISCO and is now VMware; but its been so long I'm unsure of
the details).

anyway...

i386 and amd64 are two very popular architectures which don't have
ASIDs.  Basedon prior experience, measurement of interrupt rates with
current gig-e and early 10GbE NICS, and speaking just personally: I'd
rule out that approach as a non-starter, at least as an MI solution.
Unless, that is, one opts for something like run-to-completion to
amortize the cost of the context-switch to an interrupt thread.

It's hard to go into more detail on my own opinions without recapping
various private discussions; and of course I'd need to check with the
other parties involved before doing that.


[...]

>One level may be a little extreme since you may run into trouble with
>pmap operations in drivers that need interprocessor interrupts for TLB
>shootdowns.
>Serial devices (and others) may also take a dim few on the latency
>issues this may cause.

Yes, exactly.  How to address that, whilst also keeping both important
contributors and hardware devices happy, is one of the large missing
pieces.  One might even make a case that the work done in NetBSD so
far, to improve parallelism for userspace code just might be
detrimental to improved SMP parallelism in the kernel.  I don't want
to go that far myself, though.