Re: RFC: softint-based if_input

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

On Sat, Jan 16, 2016 at 12:03 PM, Thor Lancelot Simon <tls%panix.com@localhost> wrote:
> On Fri, Jan 15, 2016 at 09:55:37PM +0000, Taylor R Campbell wrote:
>>
>> If ether_input is going to distribute packets to other CPUs at some
>> point anyway, it seems to me that it's a waste of time to do that
>> twice -- aside from the issue of IPI scarcity.  So perhaps we should
>> have either
>>
>> (a) a per-CPU pktq that never distributes packets to another CPU, or
>> (b) a single-CPU pktq, to be used only from the CPU to which the
>> device's (queue's) interrupt handler is bound.
>>
>> I suppose (a) is just a softint together with a percpu(9) of ifqs, and
>> (b) is just a softint together with an ifq.
>
> One consideration here is that with MSI and hardware packet steering
> on NICs (for example, according to a hash of the header data up to
> and including L4), the ideal situation is that you never, or almost never,
> should be distributing packets to a different CPU.  The packet should
> come in, the appropriate CPU should take the interrupt, and protocol
> processing should run to completion on that CPU -- ideally, even ensuring
> the receiving process is next scheduled on that same CPU.
>
> Solaris manages to do this much of the time, and in some configurations
> Windows does too (this is why many adapters support the "Toeplitz" hash
> Windows uses for their equivalent of our PCB lookup keys).
>
> But this is ideal.  When this isn't possible, I echo Taylor's comment
> that you don't want to move processing of a given packet between CPUs
> more than once, and I think it belongs as early in the receive path as
> possible.

Our most important target is IP forwarding so that I think
we can utilize HW multi-queue feature and interrupt distribution;
we don't need to distribution packets between CPUs by software
(at least at this point).

I think we would need software-based packet distribution when we
run NetBSD as a server, i.e., we need to deliver packets to a
different CPU where a server process which is for the packets
is running.

  ozaki-r