Re: Attempt to fix nfs client

[christos%astron.com@localhost (Christos Zoulas)]

In article <20180120144510.GA61048%atom.tmux.org@localhost>,
Tobias Ulmer  <tobiasu%tmux.org@localhost> wrote:
>TLDR: nfs_reqq crrptin, see diff below.
>
>I've been trying to track down the source of NFS client hangs and
>crashes over the last weeks, in order to get back to the things I really
>wanted to do...
>
>No special setup, the NFS Server is an OpenBSD machine serving files
>(pkgsrc, src, xsrc) from SSD (= fast replies). gbit net, udp intr
>mounts.
>
>There are any number of bug reports in gnats about this.
>1) Can't interrupt hanging process on 'intr' mount.
>
>2) Client goes dead silent and hangs despite tcpdump showing correct
>   reply from server.
>
>3) Client crashes in all places dealing with nfsreq/nfsmount pointers.
>   Mostly nfs_timer.
>
>The first was easy to fix. The main loop of nfs_receive() doesn't check
>for signals.
>
>The second issue is looping in nfs_receive, waiting for any reply to
>arrive. This is supposed to be fulfilled by nfs_timer re-sending the
>request, but investigation shows nfs_timer isn't scheduled because
>nfs_reqq is empty(!)
>
>I'm not 100% certain multiple lwps are racing to add and remove their
>requests to the global list. I could never spot contention between them
>(using mutex_trylock), but I assume they're doing so.  Once insert and
>nfsreq removal is locked, crashes were confined to nfs_timer.
>
>Before one could observe funny situations where nfs_receive() would
>return the desired reply but nfs_reply() could not find the outstanding
>request in the nfs_reqq, then discarding the valid reply as unexptected
>data.
>
>Finally, the crashes in nfs_timer. Allocating zero'ed nfsreq and
>overwriting with junk before freeing will quickly crash the machine.
>
>Sprinkling a few printfs around, it's clear that nfs_timer (callout,
>kernel_lock'ed, blocked from starting by splsoftnet) is giving up its
>lock at some point (presumably when calling into pr_send). Lwps
>will then modify and free their nfsreq in between nfs_timer
>operating on the nfs_reqq.
>
>PR 40491 (http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=40491)
>basically spells it out in one sentence, it just doesn't go far
>enough. Both nfs_reqq and nfsreq can evaporate from under you.
>
>Below is my attempt at patching this up. It seems a bit heavy handed but
>so far it's been 100% reliable (even serving nfs mounts) and perfomance is
>unchanged.  Contention is low because nfs_timer is mostly blocked from
>running by splsoftnet, and it's a recovery thing anyway.
>

That is fine as it is! Thanks for working on it, committed.

christos