Coming changes to /bin/sh

[Robert Elz <kre%munnari.OZ.AU@localhost>]

I have two changes ready to commit that I thought I would let
people know about, and solicit a few opinions as to a couple of
open issues...

The first is to implement the ";&" alternative to ";;" in case
statements - ";&" is "fallthrough", and exists in many other
shells ... actions that are connected to the following pattern
(if any) are executed after the actions for the current one (without
evaluating the pattern, or attempting to match it).

This was approved for the next (major) version of POSIX (issue 8)
back in 2011, and is (and was) implemented in most other shells
(just not ours.)   (No, I have no idea when issue8 will actually appear.)

As an exmaple of its use, the canonical example (from the forthcoming
POSIX text) is ...

x=0 y=1
case 1 in
  $((y=0)) ) ;;
  $((x=1)) ) ;&
  $((x=2)) ) echo $x.$y ;;
esac

which is required to emit "1.0" (and does with the changes I have planned,
but not yet committed).

The question here, is that bash (at least) also has a third form of
terminator to use instead of ";;" - ";;&".   That one causes fall
through to the next pattern which matches, that is, after the commands
associated with a pattern are executed, if those commands were terminated
with ";;&" then the shell will go on to look at the next pattern, evaluate
it, see if it matches, and if so, execute its commands (and repeat if it
also ends with ";;&") - if that pattern does not match, continue looking for
one that does (ie: after executing the ";;&" commands, everything continues
more or less as if the pattern which matched had not matched after all.)

I do not have an implementation of that, but I should be able to make it
happen if we want it.  The question is do we?   This one is not POSIX.


The second change is a bunch of work on ${LINENO}.   Some of you know,
I know, that the ${LINENO} implementation in our shell is something of
a hack (which nevertheless has some properties that I prefer over the
other implementations.)   But a hack it is.   In particular, to work
we must see $LINENO or ${LINENO} (or the brace form with one of the
conditional or modifier expressions as well).   In particular $((LINENO))
just gives 0 (there is not really a variable called LINENO to reference).

The changes I have ready, make LINENO into a proper variable, and I think
will make out implementation of LINENO the best that there is (all the other
shells I have tested have at least one defect in their implementations.)

The question here is what should we do with the hack?   While I haven't
tested it, I have tested enough to be fairly confident that if we just
delete it, everything will work, more or less like bash or ksh93.
But if we do, we lose one thing...

That is, as we have it now, a command like

echo "${LINENO}
${LINENO}
${LINENO}"

outputs N, N+1 and N+2 for the three instances of LINENO, neither bash
nor ksh93 do that, they either print N, three times, or N+2 three times
(ksh93 and bash resp.)   That is, a line number is associated iwth the
echo command, LINENO is set to that, and then the args are expanded,
replacing ${LINENO} with the same value every time it is encountered.

The same happens in any multi-line "string" which references ${LINENO}
(like here docs, the arg(s) to "eval", ...)

Now if it was just that, I wouldn't bother asking, "our way is better"
(NIH if you like...) but it does give rise to an anomaly, if the command
were instead ...

echo "${LINENO} $((LINENO))
${LINENO} $((LINENO))
${LINENO} $((LINENO))"

then where you'd kind of expect (perhaps hope) that each line would contain
the same number twice (and does in bash, and ksh93), with the way I have the
changes now, it does not for us - the ${LINENO} variants get expanded as
above, to the actual line number on which they appear (the hack), whereas
the $((LINENO)) forms look at the value of the LINENO variable when the
expansion is done, and so (just like the other shells) we get the same value
for each $((LINENO)) (it happens to be the number of the line containing
"echo" for us, as it is in ksh93).

So, what does everyone think?

Also, these changes will fix a bug with LINENO that I introduced
back when the here doc processing was changed, last year ... the
here doc text (in the -current sh) does not contribute to LINENO,
which continues counting just as if the here doc simply didn't exist.
(The fix could be made independanlly, it has nothing to do with the
rest of the changes - I just noticed the issue when testing all of this.)

For 99% of those of you who have read this far, stop now!

For the odd individual who really cares about the details, I am including
the script I am using to torture test this stuff...

First, here it is with line numbers (so you can see which line numbers make
sense in the output) - I will include it again, in raw form, right at the
end of the message for any of you who want to try running it on whatever
(Bourne style) shells you have available.

     1	echo "Line ${LINENO} at start"
     2	
     3	X=${LINENO}; echo "Multi-line string starts @${LINENO} ($((LINENO)))
     4	continues with line ${LINENO} ($((LINENO))) and
     5	also ${LINENO} ($((LINENO))) in a
     6	multi-line string ending at ${LINENO} ($((LINENO))) started @${X}"
     7	
     8	X=${LINENO}; echo "At start ${LINENO} ($((LINENO))) then "\
     9	\
    10	${LINENO} "($((LINENO)))" during and \
    11	\
    12	"${LINENO} ($((LINENO))) after continuation lines @ $X"
    13	
    14	X=${LINENO}; echo "Wacky weird case @$X" $\
    15	{\
    16	L\
    17	I\
    18	N\
    19	E\
    20	N\
    21	O\
    22	}
    23	
    24	X=${LINENO}; echo "Anded ${LINENO} ($((LINENO)))" &&
    25		echo "${LINENO} ($((LINENO)))" &&
    26		echo "${LINENO} ($((LINENO))) started at $X"
    27	
    28	X=${LINENO}; if echo "first in if cond @$X - ${LINENO} ($((LINENO)))"
    29		echo "and second in if test  - ${LINENO} ($((LINENO)))"
    30	then
    31		echo "${LINENO} ($((LINENO))) in then clause of if @ $X"
    32	fi
    33	
    34	X=${LINENO}; {
    35		echo "In a compound command @$X - ${LINENO} ($((LINENO)))"
    36	}
    37	
    38	X=${LINENO}; (
    39		echo "In a subshell @$X - ${LINENO} ($((LINENO)))"
    40	)
    41	
    42	X=${LINENO}; echo $(
    43		echo "in a cmdsub @$X - ${LINENO} ($((LINENO)))"
    44	)
    45	X=${LINENO}; echo `
    46		echo "in a backtick cmdsub @$X - ${LINENO} ($((LINENO)))"
    47	`
    48	
    49	X=${LINENO}; echo "In arithmetic @$X $(( LINENO )) or $(( $LINENO ))
    50	in multi-line arith $((
    51	 $LINENO +
    52	 $LINENO )) and unquoted multi-line arith:" $((
    53	 $LINENO
    54	+
    55	 $LINENO
    56	))
    57	
    58	X=${LINENO}; cat <<!
    59	In a heredoc started at $X on first line ${LINENO} ($((LINENO)))
    60	and 2nd line ${LINENO} ($((LINENO)))
    61	and last, and least: ${LINENO} ($((LINENO)))
    62	!
    63	
    64	X=${LINENO}; eval 'echo "An eval @$X line 1: ${LINENO} ($((LINENO)))"
    65				echo "Line 2: ${LINENO} ($((LINENO)))"'
    66	
    67	X=${LINENO}; . /tmp/D
    68	
    69	export LINENO; X=${LINENO}; printf 'exported LINENO @%d +1 = ' $X
    70		env | grep '^LINENO='; printf 'And from the line affer: '
    71		env | grep '^LINENO='

The file "/tmp/D" which is run as ". /tmp/D" near the end contains
just two lines ...

echo "First line in a dotfile @$X: ${LINENO}"
echo "Second and last line in dotfile: ${LINENO}"

but you could easily just delete the "." test, every shell (I tested)
handles ${LINENO} correctly in a "." file (that and the first simple
command are the only two that work the same everywhere.).  Of
course, all the other (troublesome) forms could occur in a '.' script,
but doing so would illustrate nothing extra that is useful.

The output my working-sources shell generate from this is ...

Line 1 at start
Multi-line string starts @3 (3)
continues with line 4 (3) and
also 5 (3) in a
multi-line string ending at 6 (3) started @3
At start 8 (8) then 10 (8) during and 12 (8) after continuation lines @ 8
Wacky weird case @14 22
Anded 24 (24)
25 (25)
26 (26) started at 24
first in if cond @28 - 28 (28)
and second in if test  - 29 (29)
31 (31) in then clause of if @ 28
In a compound command @34 - 35 (35)
In a subshell @38 - 39 (39)
in a cmdsub @42 - 43 (43)
in a backtick cmdsub @45 - 46 (46)
In arithmetic @49 49 or 49
in multi-line arith 103 and unquoted multi-line arith: 108
In a heredoc started at 58 on first line 59 (58)
and 2nd line 60 (58)
and last, and least: 61 (58)
An eval @64 line 1: 64 (64)
Line 2: 65 (65)
First line in a dotfile @67: 1
Second and last line in dotfile: 2
exported LINENO @69 +1 = LINENO=70
And from the line affer: LINENO=71

The values in parentheses (mostly) - that is, from $((LINENO)) are
what we would get for ${LINENO} if we remove the LINENO hack.

The output from bash (I am testing 4.3.30(1)-release -- I probably should
update it, but I doubt there have been many changes to this stuff).  It
makes no difference whether bash is running in posix mode or not.

Line 1 at start
Multi-line string starts @6 (6)
continues with line 6 (6) and
also 6 (6) in a
multi-line string ending at 6 (6) started @3
At start 10 (10) then 10 (10) during and 10 (10) after continuation lines @ 8
Wacky weird case @14 14
Anded 24 (24)
25 (25)
26 (26) started at 24
first in if cond @28 - 28 (28)
and second in if test  - 29 (29)
31 (31) in then clause of if @ 28
In a compound command @34 - 35 (35)
In a subshell @38 - 39 (39)
in a cmdsub @42 - 45 (45)
in a backtick cmdsub @45 - 48 (48)
In arithmetic @49 52 or 52
in multi-line arith 104 and unquoted multi-line arith: 104
In a heredoc started at 58 on first line 58 (58)
and 2nd line 58 (58)
and last, and least: 58 (58)
An eval @64 line 1: 65 (65)
Line 2: 66 (66)
First line in a dotfile @67: 1
Second and last line in dotfile: 2
exported LINENO @69 +1 = LINENO=69
And from the line affer: LINENO=69

Note in particular the very last test (exporting LINENO and seeing what
happens to its value, in the environment, doesn't work at all, the
exported value is just whatever LINENO was when it was exported.)

From ksh93 (the current version of ast-ksh in pkgsrc), we get

Line 1 at start
Multi-line string starts @3 (3)
continues with line 3 (3) and
also 3 (3) in a
multi-line string ending at 3 (3) started @3
At start 8 (8) then 8 (8) during and 8 (8) after continuation lines @ 8
Wacky weird case @14 14
Anded 24 (24)
25 (25)
26 (26) started at 24
first in if cond @28 - 28 (28)
and second in if test  - 29 (29)
31 (31) in then clause of if @ 28
In a compound command @34 - 35 (35)
In a subshell @38 - 39 (39)
in a cmdsub @42 - 43 (43)
in a backtick cmdsub @45 - 46 (46)
In arithmetic @49 49 or 49
in multi-line arith 98 and unquoted multi-line arith: 98
In a heredoc started at 58 on first line 58 (58)
and 2nd line 58 (58)
and last, and least: 58 (58)
An eval @64 line 1: 1 (1)
Line 2: 2 (2)
First line in a dotfile @67: 1
Second and last line in dotfile: 2
exported LINENO @69 +1 = LINENO=70
And from the line affer: LINENO=71

Here the exported LINNO works fine, but eval "string" is treated as a
script, and LINENO restarts from 1 in that string, which I don't think
is sane (that is supposed to happen in . scripts, and in functions, but
an eval'd string is neither of those.)   Bash doesn't get the line numbers
for the eval quite right (I am not sure what is happening there) but at
least they are close (off by one.)

I have tested most of the other Bourne like shells in pkgsrc, and a few
more I have access to, and all of those are even worse (and of course, the
shell distributed with NetBSD currently - all versions - is one of those.)

[You can run the script yourself with NetBSD-current's sh, don't bother
even trying using anything older.   Aside: the changes that have been made
in the past day or so are not critical to this, so a -current shell from
anytime relatively recently, since the new $(()) code was introduced, should
be OK, just the test with the var name split over multiple lines won't work
unless you have a -current shell from today (or later).]

kre

Here is the raw script...  To run it, create /tmp/D (content above) (or delete
that part of the test below), put this in a file ('x' permission not required),
and run it as "sh file" ... that's all it takes.  The first "echo" line
just below should be the first line of the file (to generate comparable results
with those above.)

echo "Line ${LINENO} at start"

X=${LINENO}; echo "Multi-line string starts @${LINENO} ($((LINENO)))
continues with line ${LINENO} ($((LINENO))) and
also ${LINENO} ($((LINENO))) in a
multi-line string ending at ${LINENO} ($((LINENO))) started @${X}"

X=${LINENO}; echo "At start ${LINENO} ($((LINENO))) then "\
\
${LINENO} "($((LINENO)))" during and \
\
"${LINENO} ($((LINENO))) after continuation lines @ $X"

X=${LINENO}; echo "Wacky weird case @$X" $\
{\
L\
I\
N\
E\
N\
O\
}

X=${LINENO}; echo "Anded ${LINENO} ($((LINENO)))" &&
	echo "${LINENO} ($((LINENO)))" &&
	echo "${LINENO} ($((LINENO))) started at $X"

X=${LINENO}; if echo "first in if cond @$X - ${LINENO} ($((LINENO)))"
	echo "and second in if test  - ${LINENO} ($((LINENO)))"
then
	echo "${LINENO} ($((LINENO))) in then clause of if @ $X"
fi

X=${LINENO}; {
	echo "In a compound command @$X - ${LINENO} ($((LINENO)))"
}

X=${LINENO}; (
	echo "In a subshell @$X - ${LINENO} ($((LINENO)))"
)

X=${LINENO}; echo $(
	echo "in a cmdsub @$X - ${LINENO} ($((LINENO)))"
)
X=${LINENO}; echo `
	echo "in a backtick cmdsub @$X - ${LINENO} ($((LINENO)))"
`

X=${LINENO}; echo "In arithmetic @$X $(( LINENO )) or $(( $LINENO ))
in multi-line arith $((
 $LINENO +
 $LINENO )) and unquoted multi-line arith:" $((
 $LINENO
+
 $LINENO
))

X=${LINENO}; cat <<!
In a heredoc started at $X on first line ${LINENO} ($((LINENO)))
and 2nd line ${LINENO} ($((LINENO)))
and last, and least: ${LINENO} ($((LINENO)))
!

X=${LINENO}; eval 'echo "An eval @$X line 1: ${LINENO} ($((LINENO)))"
			echo "Line 2: ${LINENO} ($((LINENO)))"'

X=${LINENO}; . /tmp/D

export LINENO; X=${LINENO}; printf 'exported LINENO @%d +1 = ' $X
	env | grep '^LINENO='; printf 'And from the line affer: '
	env | grep '^LINENO='