On Feb 21, 2007, at 5:57 PM, jonathan@dsg.stanford.edu wrote:

> 	(...new IPsec'd packet comes in, asserts NIC interrupt)
> 1. switch from user to NIC interrupt thread
> 	(...  NIC calls ether_input which demuxes packet, enqueues on
> 	      protocol input routine, requests softint processing, blocks)

I think eventually we won't have this push-up-the-stack's-throat model  
anymore.  A better solution is for the NIC interrupt to simply  
schedule a puller-thread to grab the packet and process it to  
completion without having to reschedule anything.

> 2. switch from NIC hardware thread to softint thread
> 	(... OCF submits job, blocks ...)
> 3. switch from softint to user	
> 	(... crypto hardware finishes, requests interrupt...)
> 4.  switch from user to crypto-interrupt thread
> 	(... crypto driver calls OCF which wakes up softint processing...)
> 5.	switch to softint thread, process cleartext packet
> 	(... done with local  kernel packet processing, softint thread  ...)
> 6.	switch back to user.
>
> [[Let's ignore, for now, whether there'd be a single softint thread,
> or a separate thread for sofnet, or multiple softnet threads.]]
>
> Is the above roughly right? If it is, then (to steal a comment I made
> to Sam Leffler early in FAST_IPSEC development): 300,000 context
> switches/sec are _not_ your friend.  And even if the NIC->softint and
> crypto->softint are a same-VM-context switch (and thus cheap, or at
> least optimizable as kthread-to-kthread), the cost of rescheduling is
> high relative to the work done per packet.  And remember, the cost is
> not just instructions, but the additional cache-footprint (and thus
> evictions) of all these switches.  Ditto for TLB state; doubly so for
> arches with software-filled TLBs.
>
> So I assume I'm missing something; can you clue me in?
>
>
>> Ok, that sounds good. How do we make this work on SPARC, m68k, and  
>> VAX?
>> Right now, interrupts-as-threads sounds like a total loss for them.  
>> If we
>> can turn it into something that's "about as good" then I think you  
>> will
>> just get a green light. :-)
>
> Bill, you mean "about as good as NetBSD is now", right?

-- thorpej