Re: On softints, softnet_lock and sleeping (aka ipv6 vs USB network interfaces)

[Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost>]

   Date: Mon, 7 Dec 2015 07:40:38 +0000
   From: Nick Hudson <skrll%netbsd.org@localhost>

   On 12/06/15 16:01, Taylor R Campbell wrote:
   > One quick fix might be to change nd6_timer to call in6_purgeaddr in a
   > workqueue (or...task, if we had that).  It seems to me that
   > in6_purgeaddr is a relatively expensive operation (and I think there's
   > a bug: it calls callout_stop via nd6_dad_stop/nd6_dad_stoptimer when
   > it should probably call callout_halt), so calling it from a callout
   > doesn't seem right anyway.

   Sure, but the "sleeping with softnet_lock held" problem still exists.

Yes -- this only solves part of your problem.  But calling
in6_purgeaddr from callout looks like a problem in itself, as does
using callout_stop in nd6_dad_stop instead of callout_halt.

   > As a practical matter, I don't see how that would help -- IPL_SOFTUSB
   > is only ever used as a mutex ipl, and IPL_SOFT* is equivalent to
   > IPL_NONE for the practical purposes of mutex(9).

   Why do IPL_SOFT* exist? If the intention is to show usage then
   IPL_SOFT is enough, right?

They can be used to block softints with splraiseipl.  For mutex(9), we
just allow the softint lwp to sleep instead of using spl* to block
soft interruption.

   > That aside, can softints even interrupt softints, or are the
   > priorities only about who goes first if two softints are scheduled
   > `simultaneously' (as far as softint_dispatch can discern)?

   The latter, iiuc.

I checked on arm, and it looks like higher-priority softints can
interrupt lower-priority softints, if I read dosoftints in
arm32_machdep.c correctly.  But I could be wrong, and on cursory
glance the story looked different for other architectures.  So it
would be nice if someone who understands softints better than I do
could swoop in and clarify, and then maybe we can update the man page
to explain the semantics more clearly.

   > If so, even then, using SOFTINT_BIO instead of SOFTINT_NET wouldn't
   > help here -- SOFTINT_BIO is even lower-priority than SOFTINT_NET, but
   > you need to allow the USB HC interrupt to run (I assume you mean,
   > e.g., ehci_softintr?) faster than SOFTINT_NET in order to wake
   > usbd_transfer while a SOFTINT_NET lwp is blocked on softnet_lock.

   Looking at spl(9) and the code again I meant SOFTINT_SERIAL. The
   idea would indeed be that the USB HC softint (e.g. ehci_softintr)
   would run before SOFTINT_NET handlers.

That would make more sense.  Worth a try, anyway.