Re: RFC: softint-based if_input

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

   Date: Fri, 15 Jan 2016 18:35:32 +0900
   From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>

   BTW, I forgot noting that ipi_sysinit in main() needed a tweak
   to use pktq_create in driver attach routines. The tweak is
   like this finally:
   http://www.netbsd.org/~ozaki-r/fix-ipi_sysinit.diff

I'm a little concerned about the number of IPIs you may introduce
here, if you create one pktq per interface, which led me to a concern
about the efficacy of distributing packets across CPUs twice.

Softints and IPIs are both scarce resources, but I think IPIs are much
more scarce: as currently set up, the kernel can have ~hundreds of
softints, but only 32 IPIs.

As far as I can tell, the only reason pktq_create establishes an IPI
is to allow executing softints on remote CPUs, so that we can
distribute the load of processing IP packets across multiple CPUs.
(Is this an advantage over just using device multiqueue support to
distribute IP packet processing?  I don't know -- rmind?)

If ether_input is going to distribute packets to other CPUs at some
point anyway, it seems to me that it's a waste of time to do that
twice -- aside from the issue of IPI scarcity.  So perhaps we should
have either

(a) a per-CPU pktq that never distributes packets to another CPU, or
(b) a single-CPU pktq, to be used only from the CPU to which the
device's (queue's) interrupt handler is bound.

I suppose (a) is just a softint together with a percpu(9) of ifqs, and
(b) is just a softint together with an ifq.