[src/trunk]: src/bin/sh Fix an old bug (very old) that was made worse in 1.12...

[kre <kre%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/13e07065652c
branches:  trunk
changeset: 996548:13e07065652c
user:      kre <kre%NetBSD.org@localhost>
date:      Mon Feb 04 09:56:26 2019 +0000

description:
Fix an old bug (very old) that was made worse in 1.128 (the "${1+$@}"
fixes) where a variable containing a CTL char (the only possibility used
to be CTLESC (0x81)) would lose that character if the variable was expanded
when "set -f" (noglob) was in effect.

1.128 made this worse by adding more 0x8z values (a couple more) which would
see the same behaviour, and one of those was noticed by Martijn Dekker.

The reasoning was that when noglob is on, when a var is expanded, there are
no magic chars, so (apparently) no need to escape anything.  Hence nothing
was escaped .. including any CTL chars that happened to be present.  When
we later rmescapes() the CTL chars that we expect might occur are summarily
removed - even if they weren't really CTL chars, but just data masquerading.

We must *always* escape any CTL char clones that are in the var value,
no matter what other conditions apply, and what we expect to happen next.

While here, fix rmescapes() (and its $(()) clone, rmescapes_nl()) to
be more robust, less likely to forget to delete anything (which was
not the issue here, just the reverse) and in a DEBUG shell, have the
shell abort() if it encounters something in rmescapes() it is not
anticipating, so the code can be made to handle it, or if it should
not happen, we can find out why it did.

XXX pullup -8 (but will need to be via patch, code is quite different).

diffstat:

 bin/sh/expand.c |  44 +++++++++++++++++++++++++++++++++++++-------
 1 files changed, 37 insertions(+), 7 deletions(-)

diffs (101 lines):

diff -r 23b069e49c93 -r 13e07065652c bin/sh/expand.c
--- a/bin/sh/expand.c   Mon Feb 04 09:54:47 2019 +0000
+++ b/bin/sh/expand.c   Mon Feb 04 09:56:26 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: expand.c,v 1.129 2018/12/03 06:41:30 kre Exp $ */
+/*     $NetBSD: expand.c,v 1.130 2019/02/04 09:56:26 kre Exp $ */
 
 /*-
  * Copyright (c) 1991, 1993
@@ -37,7 +37,7 @@
 #if 0
 static char sccsid[] = "@(#)expand.c   8.5 (Berkeley) 5/15/95";
 #else
-__RCSID("$NetBSD: expand.c,v 1.129 2018/12/03 06:41:30 kre Exp $");
+__RCSID("$NetBSD: expand.c,v 1.130 2019/02/04 09:56:26 kre Exp $");
 #endif
 #endif /* not lint */
 
@@ -938,17 +938,39 @@
                } else {
 
                        if (subtype == VSLENGTH) {
-                               for (;*val; val++)
+                               for (; *val; val++)
                                        varlen++;
                        } else if (quotes && varflags & VSQUOTE) {
+                               /*
+                                * If we are going to look for magic in the
+                                * value (quotes is set) and the expansion
+                                * occurs inside "" (VSQUOTE) then any char
+                                * that has any potential special meaning
+                                * needs to have that meaning suppressed,
+                                * so supply a CTLESC prefix for it.
+                                */
                                for (; (c = *val) != '\0'; val++) {
                                        if (NEEDESC(c))
                                                STPUTC(CTLESC, expdest);
                                        STPUTC(c, expdest);
                                }
                        } else {
-                               while (*val)
-                                       STPUTC(*val++, expdest);
+                               /*
+                                * We are going to rmescapes() later,
+                                * so make sure that any data char that
+                                * might be mistaken for one of our CTLxxx
+                                * magic chars is protected ... always.
+                                *
+                                * In BASESYNTAX only our internal CTLxxx
+                                * chars are CCTL, in other syntaxes other
+                                * chars are added (we could also use
+                                * ARISYNTAX, but this is safer)
+                                */
+                               for (; (c = *val) != '\0'; val++) {
+                                       if (BASESYNTAX[c] == CCTL)
+                                               STPUTC(CTLESC, expdest);
+                                       STPUTC(c, expdest);
+                               }
                        }
                }
        }
@@ -1968,7 +1990,7 @@
        char *p, *q;
 
        p = str;
-       while (*p != CTLESC && !IS_BORING(*p)) {
+       while (BASESYNTAX[(int)*p] != CCTL) {
                if (*p++ == '\0')
                        return;
        }
@@ -1985,6 +2007,10 @@
                }
                if (*p == CTLESC)
                        p++;
+#ifdef DEBUG
+               else if (BASESYNTAX[(int)*p] == CCTL)
+                       abort();
+#endif
                *q++ = *p++;
        }
        *q = '\0';
@@ -2007,7 +2033,7 @@
        int nls = 0, holdnl = 0, holdlast;
 
        p = str;
-       while (*p != CTLESC && !IS_BORING(*p)) {
+       while (BASESYNTAX[(int)*p] != CCTL) {
                if (*p++ == '\0')
                        return;
        }
@@ -2031,6 +2057,10 @@
                }
                if (*p == CTLESC)
                        p++;
+#ifdef DEBUG
+               else if (BASESYNTAX[(int)*p] == CCTL)
+                       abort();
+#endif
 
                holdlast = holdnl;
                holdnl = is_in_name(*p);        /* letters, digits, _ */