Re: RFC: softint-based if_input

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

   Date: Fri, 29 Jan 2016 12:44:07 +0900
   From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>

   So here is a complete patch:
   http://www.netbsd.org/~ozaki-r/softint-if_input-full.diff

   With the patch if_input and above should not run in HW
   interrupt anymore.

Instead of adding fields to the already overpopulated struct ifnet,
how about creating a new struct if_percpuq or something which drivers
can voluntarily use if they desire?

struct if_percpuq {
	void		*ipq_si;
	struct percpu	*ipq_ifqs;	/* struct ifqueue */
};

struct if_percpuq *
	if_percpuq_create(void (*)(void *), void *);
void	if_percpuq_destroy(struct if_percpuq *);
void	if_percpuq_enqueue(struct if_percpuq *, struct mbuf *);

To check at compile-time to make sure all callers have been converted
to the softint-only ifp->if_input, we could change its signature to
add a dummy argument, or rename the member.  The signature/name of
ifp->if_input is also closer to the real change -- forbidding *_input
in hardintr context -- than the signature of if_initialize.

The reason I suggest a separate API which drivers can voluntarily use
is that I expect we may want alternatives -- e.g., drivers without
multiqueue support may want to use a pktq instead and automatically
distribute to other CPUs[*].  I've also never liked initialization
flags like IF_INPUTF_NO_SOFTINT to suppress a newly-default option.

Also: I think your patch is broken as is for all USB NICs -- all their
`interrupt' routines actually run in softint context, if I'm not
mistaken, so this kassert will fire:

+void
+if_input(struct ifnet *ifp, struct mbuf *m)
+{
+
+	if (ifp->if_input_ifqs) {
+		struct ifqueue *ifq = percpu_getref(ifp->if_input_ifqs);
+
+		KASSERT(cpu_intr_p());

The same may go for some other NICs, such as se(4), for which there is
a possible use from thread context which I can't rule out in thirty
seconds.  Another possible case is xennet(4).


[*] Currently this happens (or could happen) with ip_pktq anyway, but
if I recall correctly, on discussion, rmind@ agreed with me that
that's the wrong place and because of variation in hardware support
for rx multiqueue or hashing flows, it should be the NIC driver's job
to decide how to distribute packets -- even if that means explicitly
calling a software abstraction to make the decision when the NIC can't
do rx multiqueue or hash flows itself.