Re: RFC: softint-based if_input

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

On Sat, Jan 16, 2016 at 6:55 AM, Taylor R Campbell
<campbell+netbsd-tech-kern%mumble.net@localhost> wrote:
>    Date: Fri, 15 Jan 2016 18:35:32 +0900
>    From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>
>
>    BTW, I forgot noting that ipi_sysinit in main() needed a tweak
>    to use pktq_create in driver attach routines. The tweak is
>    like this finally:
>    http://www.netbsd.org/~ozaki-r/fix-ipi_sysinit.diff
>
> I'm a little concerned about the number of IPIs you may introduce
> here, if you create one pktq per interface, which led me to a concern
> about the efficacy of distributing packets across CPUs twice.
>
> Softints and IPIs are both scarce resources, but I think IPIs are much
> more scarce: as currently set up, the kernel can have ~hundreds of
> softints, but only 32 IPIs.

Oh, I didn't know the scarceness.

>
> As far as I can tell, the only reason pktq_create establishes an IPI
> is to allow executing softints on remote CPUs, so that we can
> distribute the load of processing IP packets across multiple CPUs.

I think so.

> (Is this an advantage over just using device multiqueue support to
> distribute IP packet processing?  I don't know -- rmind?)
>
> If ether_input is going to distribute packets to other CPUs at some
> point anyway, it seems to me that it's a waste of time to do that
> twice -- aside from the issue of IPI scarcity.  So perhaps we should
> have either
>
> (a) a per-CPU pktq that never distributes packets to another CPU, or
> (b) a single-CPU pktq, to be used only from the CPU to which the
> device's (queue's) interrupt handler is bound.
>
> I suppose (a) is just a softint together with a percpu(9) of ifqs, and
> (b) is just a softint together with an ifq.

Actually the reason I used pktq was that it's the easiest way to
write a POC code for softint + percpu queues, and I didn't have
plan to utilize pktq's packet distribution feature. So not using
pktq in favor of HW multi-queue support is okay for me (for us),
at least at an first step of this work.

I'll rewrite the patch as your suggestion (I prefer (a) for now).

Thanks,
  ozaki-r