Re: RFC: softint-based if_input

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

   Date: Wed, 27 Jan 2016 16:51:22 +0900
   From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>

   Here it is: http://www.netbsd.org/~ozaki-r/softint-if_input-ifqueue.diff

   Results of performance measurements of it are also added to
   https://gist.github.com/ozaki-r/975b06216a54a084debc

   The results are good but bothers me; it achieves better performance
   than vanilla (and the 1st implementation) on high load (IP forwarding).
   For fast forward, it also beats the 1st one.

   I thought that holding splnet during ifp->if_input (splnet is needed
   for ifqueue operations and so keep holding in the patch) might affect
   the results. So I tried to release during ifp->if_input but the results
   didn't change so much (the result of IP forwarding is still better
   than vanilla).

   Anyone have any ideas?

Here's a wild guess: with vanilla, each CPU does

	wm_rxeof loop iteration
	if_input processing
	wm_rxeof loop iteration
	if_input processing
	...

back and forth.  With softint-rx-ifq, each CPU does

	wm_rxeof loop iteration
	wm_rxeof loop iteration
	...
	if_input processing
	if_input processing
	...

because softint processing is blocked until the hardintr handler
completes.  So vanilla might make less efficient use of the CPU cache,
and vanilla might leave the rxq full for longer so that the device
cannot fill it as quickly with incoming packets.

Another experiment that might be worthwhile is to bind the interrupt
to a specific CPU, and then use splnet instead of WM_RX_LOCK to avoid
acquiring and releasing a lock for each packet.  (On Intel >=Haswell,
we should use transactional memory to avoid bus traffic for that
anyway (and maybe invent an MD pcq(9) that does the same).  But the
experiment with wm(4) is easier, and not everyone has transactional
memory.)