Redoing the code in /bin/sh to handle the issues in PR bin/48875

[Robert Elz <kre%munnari.OZ.AU@localhost>]

Some (or many) of you may be aware (and if not, please read the PR)
that the code added in 2016 to deal with PR bin/48875 was never really
correct - it just had not, until recently, caused anyone any problems
so it had just been sitting there, doing its thing, and bothering no-one
really, until just recently, when it actually did break a script.

Thus I am going to (totally) revert the "fix" that was installed, and instead
do something different (which would have eventually happened anyway).
Rather than attempting to "fix" the PR (which I suspect is not really possible
in the general case) the new code will simply reduce the chances of the
delays which were the subject of the complaint in the PR from occurring.
It is essentially a "performance enhancement" (which I have been resisting
working on until all the known bugs are first fixed) which creates less
sub-shells (the shell forks much less) almost all stolen from the FreeBSD
shell (so the basic scheme has already had lots of testing there.)  Because
there are less sub-shells, there is less opportunity for one of them to be
floating around holding a file descriptor open, which was the genesis of
the 48875 problem.

Part of this update will be to add some text to sh(1) to explain things a little
better.   The point of this e-mail is to seek assistance with getting that
text into an acceptable form - while English is my native language, it
is not the one I prefer to write (and I claim no proficiency at all).

The following is my current proposal to add as a new paragraph,
to the end of the current section in sh(1) under the heading
"Command Substitution".  This paragraph, is as long, or a bit longer
than all that is currently there under that heading (which is going to
get some minor wording changes as well, but they aren't a problem) -
and so might be (!!?) a bit over the top.

I'd appreciate any suggestions for improvements, particularly ways
to convey the information included in a more succinct form.

This is the (new) text I currently have ...

     Note that if a command substitution includes commands to be run in the
     background, the sub-shell running those commands will only wait for them
     to complete if an appropriate wait command is included in the command
     list.  However, the shell in which the result of the command substitution
     will be used is waiting for both the sub-shell to exit, and for the file
     descriptor that was initially standard output for the command
     substitution sub-shell to be closed.  While the exit of the sub-shell
     closes its standard output, any background process left running may still
     have that file descriptor open.  This includes yet another sub-shell
     which might have been (almost invisibly) created to wait for some other
     command to complete, and even where that other command has had its
     standard output redirected or closed, the waiting sub-shell may still
     have it open.  Thus there is no guarantee that the result of a command
     substitution will be available in the shell which is to use it before all
     of the commands started by the command substitution have completed,
     though careful coding can often avoid delays beyond the termination of
     the command substitution sub-shell.

Suggestions?

Thanks,

kre

ps: the intended new code runs the example in the PR without a delay,
and passes all the sh ATF tests (including the even more difficult tests
added to detect whether the PR is "fixed" or not).   It also (as best I can
tell) does not break pkg_chk -av ...   I am still doing more tests though.