Re: RFC: softint-based if_input

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

On Mon, Jan 25, 2016 at 1:06 PM, Taylor R Campbell
<campbell+netbsd-tech-kern%mumble.net@localhost> wrote:
>    Date: Mon, 25 Jan 2016 11:25:16 +0900
>    From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>
>
>    On Tue, Jan 19, 2016 at 2:22 PM, Ryota Ozaki <ozaki-r%netbsd.org@localhost> wrote:
>    (snip)
>    >> (a) a per-CPU pktq that never distributes packets to another CPU, or
>    >> (b) a single-CPU pktq, to be used only from the CPU to which the
>    >> device's (queue's) interrupt handler is bound.
>    >>
>    > I'll rewrite the patch as your suggestion (I prefer (a) for now).
>
>    Through rewriting it, I feel that it seems to be a lesser version of
>    pktqueue. So I think it may be better changing pktqueue to have a flag
>    to not distribute packets between CPUs than implementing another one
>    duplicating pktqueue. Here is a patch with the approach:
>    http://www.netbsd.org/~ozaki-r/pktq-without-ipi.diff
>
>    If we call pktq_create with PKTQ_F_NO_DISTRIBUTION, pktqueue doesn't
>    setup IPI for softint and never call softint_schedule_cpu (i.e.,
>    never distribute packets).
>
>    How about the approach?
>
> Some disjointed thoughts:
>
> 1. I don't think you actually need to change pktq(9).  It looks like
> if you pass in cpu_index(curcpu()) for the hash, it will consistently
> use the current CPU, for which softint_schedule_cpu has a special case
> that avoids ipi.  So I don't expect it's substantially different from
> <https://www.netbsd.org/~ozaki-r/softint-if_input.diff> -- though
> maybe measurements will show my analysis is wrong!

My intention is to prevent ipi_register in pktq_create and
so we don't need ipi_sysinit movement...

>
> 2. Even though you avoid ipi(9), you're still using pcq(9), which
> requires interprocessor synchronization -- but that is an unnecessary
> cost because you're simply passing packets from hardintr to softintr
> context on a single CPU.  So that's why I specifically suggested ifq,
> not pcq or pktqueue.

...though, right. membars in pcq(9) are just overhead.

Okay, I'll implement softint + percpu irqs.

>
> 3. Random thought: If we do polling, I wonder whether instead of (or
> in addition to) polling for up to (say) 100 packets in a softint, we
> really ought to poll for arbitrarily many packets in a kthread with
> KTHREAD_TS, so that we don't need to go back and forth between
> hardintr/softintr during high throughput, but we also don't starve
> user threads in that case.

Actually that was a POC implementation just to measure how polling
is efficient (or not). So I don't intend to use the implementation
as it is.

>
> I seem to recall starvation of user threads is what motivated matt@ to
> split packet processing between a softint and a workqueue, depending
> on the load, in bcmeth(4) (sys/arch/arm/broadcom/bcm53xx_eth.c).
> Maybe he can comment on this?  Have you studied how this driver works,
> and maybe pq3etsec(4) too, which also does polling?

I had read pq3etsec(4) but not bcmeth(4). pq3etsec(4) seems to use
only softint.

Anyway I also concerned user threads starvation during implementing
polling on wm(4). So the combination use of softint and workqueue
sounds good. (FreeBSD's igb driver also does a similar technique,
IIUC.)

I (or someone) will work on implementing such mechanism on wm(4)
(I guess not soon though...). I think it is better to provide
a common framework of polling mode for drivers if possible.

Thanks,
  ozaki-r