In message <20070221234224.GF26468@netbsd.org>,
Bill Studenmund writes:

>On Wed, Feb 21, 2007 at 10:51:03PM +0000, Andrew Doran wrote:

>> What I want to do will on x86 add 29 arithmetic instructions to the
>> interrupt path (as of now). That's means in the common, non blocking case,
>> it's ~free. I'd be pretty bummed if we undo some of that and the reasoning
>> for doing so is to make the system a better fit for ~20 year old systems.

Andrew, just what will trigger the blocking (expensive) case?
And how  expensive is it?

Can I offer the following example? Suppose for the sake of discussion,
that I'm running IPsec traffic with FAST_IPSEC, and handling a modest
50,000 packets per second.  Suppose further that there's also always
userland processes ready to run.

My first guess (perhaps overly naive) on how thread-per-interrupt
would work is something like this:

	(...new IPsec'd packet comes in, asserts NIC interrupt)
1. switch from user to NIC interrupt thread
	(...  NIC calls ether_input which demuxes packet, enqueues on
	      protocol input routine, requests softint processing, blocks)
2. switch from NIC hardware thread to softint thread
	(... OCF submits job, blocks ...)
3. switch from softint to user	
	(... crypto hardware finishes, requests interrupt...)
4.  switch from user to crypto-interrupt thread
	(... crypto driver calls OCF which wakes up softint processing...)
5.	switch to softint thread, process cleartext packet
	(... done with local  kernel packet processing, softint thread  ...)
6.	switch back to user. 

[[Let's ignore, for now, whether there'd be a single softint thread,
or a separate thread for sofnet, or multiple softnet threads.]]

Is the above roughly right? If it is, then (to steal a comment I made
to Sam Leffler early in FAST_IPSEC development): 300,000 context
switches/sec are _not_ your friend.  And even if the NIC->softint and
crypto->softint are a same-VM-context switch (and thus cheap, or at
least optimizable as kthread-to-kthread), the cost of rescheduling is
high relative to the work done per packet.  And remember, the cost is
not just instructions, but the additional cache-footprint (and thus
evictions) of all these switches.  Ditto for TLB state; doubly so for
arches with software-filled TLBs.

So I assume I'm missing something; can you clue me in?


>Ok, that sounds good. How do we make this work on SPARC, m68k, and VAX?
>Right now, interrupts-as-threads sounds like a total loss for them. If we
>can turn it into something that's "about as good" then I think you will
>just get a green light. :-)

Bill, you mean "about as good as NetBSD is now", right?