shell (/bin/sh) pattern matching bugs

[Robert Elz <kre%munnari.OZ.AU@localhost>]

Hi all,

I am soon going to add an ATF test for sh pattern matching (it tests all
3 forms, glob expansion, case patterns, and ${var#patern})

There are many tests, some of them quite esoteric...

First a general heads up ... the glob tests need files to match
against, and many of them ... so the test for that creates > 17000
files (all empty - other than the directories to contain them)
which means that the filesystem on which the ATF temporary
directories are made needs to have around 18000 free inodes
before the test starts (and about 1.5MiB free space for the
directories to contain the empty files).

That isn't the primary point of this mail, as I don't expect it to be
a real issue for most people (any average sized tmpfs should have
no real issue with that) - rather before committing the tests (which
currently fail) I wanted to give those of you who understand how
sh pattern matching really works a chance to challenge my interpretation
of what the results should be for the sub-tests that fail.

Note that all of the issues relate to quoting in one form or another,
so what is really in issue is how quoting and pattern matching interact.

Aside: some of this is in areas where the POSIX spec is either unclear
or simply wrong .. there has been discussion on relevant lists related to
fixing that, and eventually that will happen ... but it does mean that
simply going to the current posix spec and reading it will not necessarily
help a lot.

Another aside: the "case [ in [[:alpha:]]) ..." bug that was fixed a few days ago
in current was detected by these tests - but that one was a simple obvious sh
coding bug (copied from FreeBSD when the code to implement that kind of
thing was imported ... and yes, I did tell the FreeBSD people about it, and
their shell is also now fixed -- they actually committed the fix well before I did.)

Below I include the 14 (of 261) case pattern tests that fail in a sh from HEAD.

These are from an ATF run that includes the (currently uncommitted) new test.
The filename expansion (glob) tests have similar issues, but cannot easily be
demonsrated in an e-mail like this as without the 17000 files to see what 
should match, they are meaningless...   The var expansion pattern matching
I have not really completed writing yet (it doesn't test as much as the others,
and I am not sure when/if I ever will).   Hence only the failing case tests are here...

Note: I hand edited the ATF output to avoid noise and confusion, I think I left
the essence untouched, but ...

So below please find 14 pairs of lines (with empty lines between).   The [NNN] 
that prefixes each line is the sub-test number which is currently meaningful to
no-one but me, but here we can use those to refer to specific tests.

In each pair of lines, the first is the command that was run, and the second
says what happened and what was expected -- since all of the case test commands
either echo 'M' (when the pattern matches) or 'X' when it does not, and since these
are only the failing cases - the second line just says that M was expected and
X was obtained, or vice versa - so it gets a bit boring...

I'd appreciate it if you could look over these tests (and please consider them
carefully...) and see if you disagree with the results that the test expects.

If I have any of them wrong, I can fix them before committing the test.
Once the test is there, I will start committing fixes to sh so that the tests
eventually pass (I doubt any of these bugs are serious enough to
delay NetBSD-8 - just maybe #51 - they can be corrected in 8.0.1 or something.)

Thanks,

kre

[51] case "[a-c]" in ([a-c\]) printf M;; (*) printf X;; esac
[51] Expected output 'M', received 'X'

[97] var="[:alpha:]"; case "[" in (["$var"]) printf M;; (*) printf X;; esac
[97] Expected output 'M', received 'X'

[131] case "[x?abc" in ([[-\]]?\?*) printf M;; (*) printf X;; esac
[131] Expected output 'M', received 'X'

[132] case "]x?abc" in ([[-\]]?\?*) printf M;; (*) printf X;; esac
[132] Expected output 'M', received 'X'

[133] case "\\x?abc" in ([[-\]]?\?*) printf M;; (*) printf X;; esac
[133] Expected output 'M', received 'X'

[136] case "[]?" in ([[-\]]?\?*) printf M;; (*) printf X;; esac
[136] Expected output 'M', received 'X'

[147] var='\z'; case ${var} in (${var}) printf M;; (*) printf X;; esac
[147] Expected output 'X', received 'M'

[148] var='\z'; case ${var} in ("${var}") printf M;; (*) printf X;; esac
[148] Expected output 'X', received 'M'

[151] IFS=?; set -- a c; case abc in ("$*") printf M;; (*) printf X;; esac
[151] Expected output 'X', received 'M'

[152] IFS=?; set -- a c; case "a c" in ("$*") printf M;; (*) printf X;; esac
[152] Expected output 'X', received 'M'

[155] IFS=*; set -- a c; case abc in ("$*") printf M;; (*) printf X;; esac
[155] Expected output 'X', received 'M'

[156] IFS=*; set -- a c; case "a c" in ("$*") printf M;; (*) printf X;; esac
[156] Expected output 'X', received 'M'

[168] IFS=-; set -- a c; case b in (["$*"]) printf M;; (*) printf X;; esac
[168] Expected output 'X', received 'M'

[170] IFS=-; set -- a c; case - in (["$*"]) printf M;; (*) printf X;; esac
[170] Expected output 'M', received 'X'