Re: RFC: softint-based if_input

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Fri, Jan 15, 2016 at 09:55:37PM +0000, Taylor R Campbell wrote:
> 
> If ether_input is going to distribute packets to other CPUs at some
> point anyway, it seems to me that it's a waste of time to do that
> twice -- aside from the issue of IPI scarcity.  So perhaps we should
> have either
> 
> (a) a per-CPU pktq that never distributes packets to another CPU, or
> (b) a single-CPU pktq, to be used only from the CPU to which the
> device's (queue's) interrupt handler is bound.
> 
> I suppose (a) is just a softint together with a percpu(9) of ifqs, and
> (b) is just a softint together with an ifq.

One consideration here is that with MSI and hardware packet steering
on NICs (for example, according to a hash of the header data up to
and including L4), the ideal situation is that you never, or almost never,
should be distributing packets to a different CPU.  The packet should
come in, the appropriate CPU should take the interrupt, and protocol
processing should run to completion on that CPU -- ideally, even ensuring
the receiving process is next scheduled on that same CPU.

Solaris manages to do this much of the time, and in some configurations
Windows does too (this is why many adapters support the "Toeplitz" hash
Windows uses for their equivalent of our PCB lookup keys).

But this is ideal.  When this isn't possible, I echo Taylor's comment
that you don't want to move processing of a given packet between CPUs
more than once, and I think it belongs as early in the receive path as
possible.

-- 
  Thor Lancelot Simon	                                     tls%panix.com@localhost

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."      - H.L.A. Hart