Re: kern/59175: posix_spawn hang, hanging other process too

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

So here's what I think is happening:

1. do_posix_spawn in the parent p1 creates a process p2 with
   proc_alloc, which has:
   - p2->p_stat = SIDL (can't be looked up with proc_find)
   - p2->p_vmspace = &vmspace0 (initial vmspace, kernel's) or NULL
   - p2->p_psstrp = garbage from recycled struct proc or NULL

   In particular, if p2 is freshly allocated, it will have null
   p_vmspace and p_psstrp; if it was recycled, p_vmspace will have
   been set to proc0.p_vmspace=&vmspace0 shortly before recycling
   in uvm_proc_exit:
   https://nxr.netbsd.org/xref/src/sys/uvm/uvm_glue.c?r=1.182#414
   And p_psstrp will have been left untouched.

2. do_posix_spawn in the parent creates a lwp in p2 to run
   spawn_return, sets p2->p_stat = SACTIVE (so it _can_ be looked up
   with proc_find), makes the lwp runnable, and waits for it to ack
   (sed_cv_child_ready).

3. spawn_return in the child p2 acks immediately (sed_child_cv_ready)
   because neither POSIX_SPAWN_RETURNERROR nor POSIX_SPAWN_SETPGROUP
   is set.

4. do_posix_spawn in the parent returns the child pid to the caller.

5. parent tries sysctl kern.proc_args.<childpid>.argv or something
   which takes p2->p_reflock and does copyin_psstrings(p2, ...) which
   copies in p2->p_psstrp (garbage from recycled struct proc) from
   p2->p_vmspace (&vmspace0, kernel's).

   I'm not sure exactly what will happen here but it seems to send the
   kernel into a loop; in any case, whatever the symptom is, nothing
   good can come of attempting copyin_psstrings(p2, ...) before
   p2->p_vmspace and p2->p_psstrp are initialized.

6. spawn_return in the child waits for p2->p_reflock before it can
   call execve_runproc to initialize p2->p_vmspace and p2->p_psstrp.

In step (5), the parent gets stuck in a loop holding p2->p_reflock
because p2->p_vmspace and p2->p_psstrp aren't initialized, while
concurrently in step (6) the child is stuck waiting for p2->p_reflock
before it can initialize p2->p_vmspace and p2->p_psstrp -- deadlock.

(The window of time between (3) and (6) is pretty small, so the parent
-- or any other process like a concurrent htop(1) -- has to be pretty
quick about invoking sysctl during that window to win the race.)


I think the general problem here is that the pid should not be usable
by anything outside (like ps(1) or top(1), via sysctl kern.proc_args
and stuff, in turn via proc_find) until execve_runproc has fully
initialized the vmspace and psstrp.

But I don't know if the child process can even run before p2->p_stat
is set to SACTIVE, at which point it can be looked up with proc_find.
Maybe we can safely defer the transition SIDL -> SACTIVE to
spawn_return, if the lwp can run even if the process is not SACTIVE.
And we should audit fork1 vs do_posix_spawn to make sure everything
that fork1 initializes is also initialized by do_posix_spawn by the
time of the SIDL -> SACTIVE transition.

Separately, the parent can't return the child pid to the caller until
that's done and the pid is usable (e.g., for waitpid(2) or kill(2)).
So the early parent-wakeup path in spawn_return is probably just not
workable.