Re: sh(1) vs. ksh(1)

[Alan Barrett <apb%cequrux.com@localhost>]

On Fri, 24 Oct 2008, Patrick Welche wrote:
> > I am not sure that the meaning of <backslash><doublequote> inside
> > backquotes inside double quotes is well defined.  The NetBSD ksh(1) man
> > page says that it behaves differently depending on whether or not pOSIX
> > mode is in effect.  See these extracts from the ksh(1) man page: 
> 
> cf bin/26493

NetBSD ksh behaves just like pdksh here (varying its interpretation of

        echo "`echo \"hi\"`"

according to whether "set -o posix" or "set +o posix" is in effect.

ksh93 doesn't even have a "set -o posix" mode, and behaves in the
sensible way.

Comments in src/usr.bin/ksh/lex.c quote parts of
some version of the POSIX specification.  The
wording quoted is very similar to that in SUSv3
<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
but the authors of pdksh apparently interpreted the wording differently
from everybody else.  To me, it seems clear that the words "having
escape characters removed" mean, in that context, that the backslash
before the double quote in "...`...\"...`..." is removed as part of
the handling of the outer double quotes, so that the backslash is not
present when the inner backquotes are handled.

Are there any objections to the following patch, which makes NetBSD
ksh handle "...`...\"...`..." in the same way regardless of posix mode
or non-posix mode?  To make the patch readable, I have not changed the
indentation of the inside of the if...{...} block.  If the patch is
acceptable then I would change the indentation before committing.)

--apb (Alan Barrett)

Index: bin/ksh/ksh.Man
===================================================================
--- bin/ksh/ksh.Man     30 Jul 2007 18:01:42 -0000      1.20
+++ bin/ksh/ksh.Man     25 Oct 2008 19:50:40 -0000
@@ -479,8 +479,12 @@
 followed by a newline, both the \fB\e\fP and the newline are stripped;
 otherwise, both the \fB\e\fP and the character following are unchanged.
 .PP
-Note: see POSIX Mode below for a special rule regarding sequences
-of the form \fB"\fP...\fB`\fP...\fB\e"\fP...\fB`\fP..\fB"\fP.
+Note: An earlier version of ksh(1) changed the interpretation of sequences
+of the form \fB"\fP...\fB`\fP...\fB\e"\fP...\fB`\fP..\fB"\fP
+according to whether or not POSIX mode was in effect.
+In the current implementation, the backslash in \fB\e"\fP
+is seen and removed by the outer \fB"\fP...\fB"\fP,
+so the backslash is not seen by the inner \fB`\fP...\fB`\fP.
 .\"}}}
 .\"{{{  Aliases
 .SS "Aliases"
@@ -1531,15 +1535,16 @@
 reading of \fB$ENV\fP: if not in posix mode, the \fBENV\fP parameter
 is not expanded and included when the shell starts.
 .sh)
-.IP \ \ \(bu
-\fB\e"\fP inside double quoted \fB`\fP..\fB`\fP command substitutions:
-in posix mode, the \fB\e"\fP is interpreted when the command is interpreted;
-in non-posix mode, the backslash is stripped before the command substitution
-is interpreted.
-For example, \fBecho "`echo \e"hi\e"`"\fP produces `"hi"' in
-posix mode, `hi' in non-posix mode.
-To avoid problems, use the \fB$(...\fP)
-form of command substitution.
+.\" The following behaviour is not useful and has been removed in NetBSD
+.\" .IP \ \ \(bu
+.\" \fB\e"\fP inside double quoted \fB`\fP..\fB`\fP command substitutions:
+.\" in posix mode, the \fB\e"\fP is interpreted when the command is 
interpreted;
+.\" in non-posix mode, the backslash is stripped before the command 
substitution
+.\" is interpreted.
+.\" For example, \fBecho "`echo \e"hi\e"`"\fP produces `"hi"' in
+.\" posix mode, `hi' in non-posix mode.
+.\" To avoid problems, use the \fB$(...\fP)
+.\" form of command substitution.
 .IP \ \ \(bu
 \fBkill \-l\fP output: in posix mode, signal names are listed one a single 
line;
 in non-posix mode, signal numbers, names and descriptions are printed in
Index: bin/ksh/lex.c
===================================================================
--- bin/ksh/lex.c       11 Sep 2005 22:16:00 -0000      1.12
+++ bin/ksh/lex.c       25 Oct 2008 19:50:41 -0000
@@ -334,26 +334,13 @@
                                *wp++ = COMSUB;
                                /* Need to know if we are inside double quotes
                                 * since sh/at&t-ksh translate the \" to " in
-                                * "`..\"..`".
-                                * This is not done in posix mode (section
-                                * 3.2.3, Double Quotes: "The backquote shall
-                                * retain its special meaning introducing the
-                                * other form of command substitution (see
-                                * 3.6.3). The portion of the quoted string
-                                * from the initial backquote and the
-                                * characters up to the next backquote that
-                                * is not preceded by a backslash (having
-                                * escape characters removed) defines that
-                                * command whose output replaces `...` when
-                                * the word is expanded."
-                                * Section 3.6.3, Command Substitution:
-                                * "Within the backquoted style of command
-                                * substitution, backslash shall retain its
-                                * literal meaning, except when followed by
-                                * $ ` \.").
+                                * "`..\"..`".  POSIX also requires this.
+                                * An earlier version of ksh misinterpreted
+                                * the POSIX specification and performed
+                                * removal of backslash escapes only if
+                                * posix mode was not in effect.
                                 */
                                statep->ls_sbquote.indquotes = 0;
-                               if (!Flag(FPOSIX)) {
                                        Lex_state *s = statep;
                                        Lex_state *base = state_info.base;
                                        while (1) {
@@ -369,7 +356,6 @@
                                                        break;
                                                base = s-- - STATE_BSIZE;
                                        }
-                               }
                                break;
                          default:
                                *wp++ = CHAR, *wp++ = c;