workqueue for pr_input

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

Hi,

I propose a controversial change that makes some
pr_input workqueue: icmp_input, icmp6_input and
carp_input.

The reason to do so is that they may add, delete
and update IP addresses and rtentries. If NET_MPSAFE
is enabled, deleting and updating such data require
psz/psref waits (pserialize_perform and
psref_target_destroy) that sleep and cannot be used
in softint.

Dealing with deleting data may have another option
that defers only a destruction into workqueue, not
making whole pr_input workqueue, as rt_free does
for now. However, for updating data, we cannot use
the same approach.

Merits of this approach are:
- Every additions/deletions/updates on IP addresses
  and rtentries will be done in normal LWP contexts,
  neither in softint nor in hardware interrupt.
  - It also pulls all malloc/free out of softint and
    we can make them M_WAITOK or KM_SLEEP.
  - In reality, tcp_timer_rexmt is a remaining callout
    that can delete rtentries, but we can fix in the
    future.
- The logic of destructions/updates of addresses/rtentries
  don't need to take care of execution contexts.
  - We can get rid of complex deferred rt_free.
- We would be able to replace rwlocks for L2 caches
  (lltable/llentry) with psz/psref.
  - Though I'm not sure psz/psref suit for them.

The demerit is that that mechanism adds non-trivial
overhead; RTT of ping increases by 30 usec.

Here is a patch of the approach applied to icmp6_input:
  http://www.netbsd.org/~ozaki-r/workqueue-icmp6_input.diff

This is icmp6_input specific, but if we apply the approach
to icmp_input and carp_input as well, we would be able to
implement a common framework on struct protosw.

Note that with the patch all ATF tests for networking pass
even if NET_MPSAFE is enabled in rump kernels.

Any comments, suggestions, or objections?

Thanks,
  ozaki-r