Reorganizing src/tests

[Julio Merino <jmmv84%gmail.com@localhost>]

Hello,

[ Disclaimer: this is a perfect thread for a bikeshed.  So stop
reading right now if you do not use tests, do not care about them
and/or hate them. ]

This is a proposal to reorganize the contents of src/tests to address
two major concerns:

A) Tests should be alongside the sources they test.  This gives tests
more visibility and easier access, like manual pages, so it will
hopefully make people more responsible about keeping them up to date.
B) The structure of src/tests is confusing and appears to be
arbitrary.  (It is not: it follows the syspkgs naming scheme.  And if
you want to complain about this, it's too late: I made a proposal
about this a few years ago and got no complains at that time.)

I think we all (those that care about the tests, that is, which is
currently 4, 5 people?) agree that the two points above are major
concerns with the current src/tests hierarchy.

Now, to address this point we must consider two things: the layout of
the *source* of the tests, and the layout of the *installed* tests.
They should be as similar to each other as possible to minimize
confusion.

Tests location
**************

Some tests can go right next to the program they are testing whereas
others can't.  For example, it's easy to see that tests for cp(1) go
into src/bin/cp/ and get installed into /usr/tests/bin/cp/.  But it's
harder to know where to put tests that exercise several components.
Solving this is as easy as following the layout of manual pages: put
some next to the programs and put those that can't be classified under
src/tests/ (to be clear, I'm referring to src/share/man/)

I've heard some suggestion to put tests in a 'tests/' subdirectory, so
you'd have src/bin/cp/tests/.  I think that's a bad idea for two
reasons:

1) If we have this source layout and want to match the installed
layout, then the tests have to get installed under
/usr/tests/bin/cp/tests/, which is silly.  If we put the tests in
src/bin/cp/tests/ and install them in /usr/tests/bin/cp/, then we are
losing consistency which I think is a bad thing.  (Note that tests can
query what their "source directory" is, and such inconsistency will
cause lots of confusion.)

2) By introducing the tests/ subdirectory under each other directory,
we are still hiding tests.  E.g. if I'm editing cp.c and I see right
next to it a file called cp_test.c, I'll be tempted to edit it.  If
cp_test.c is under tests/cp_test.c, I can easily miss it.  Note that
we don't put manual pages in a man/ subdirectory, and for a good
reason.  Oh, and directories in cvs are expensive (braindead?).

Some tests come with data files, though, and I understand that these
can really clutter the source directory.  Hiding them under a
subdirectory 'testdata' sounds fine.

So the proposed scheme is:

- Put test source files right next to the source file they are
testing.  E.g. src/bin/cp/cp.c, src/bin/cp/cp_test.c.
- Put auxiliary data files under a testdata/ subdirectory.  E.g.
src/bin/cp/testdata/file1, src/bin/cp/testdata/file2.
- For tests that can't be easily put next to code, put them somewhere
under src/tests.

And the installed files would be:

/usr/tests/bin/cp/cp_test
/usr/tests/bin/cp/testdata/file1
/usr/tests/bin/cp/testdata/file2

Naming scheme
*************

Current test programs are prefixed by 't_'.  This is a carry-over from
atf's original naming scheme, which I now think was stupid.  Such a
prefix is cryptic and also hides tests in directory listings because
they do not appear right next to the source file.

Given that we'll be "moving" files around cvs, we can as well take the
chance to rename them.  I've come to find the _test suffix valuable,
because then both source/tests appear right next to each other in
directory listings.  (That's why I've been using cp_test instead of
t_cp in the examples above.)

So my proposal is to rename the test programs from t_foo to foo_test,
only if/when they get moved in cvs.  For auxiliary data files, which
were previously prefixed by h_, we just drop the prefix altogether
because we are putting these under testdata/.

Consider the following subdirectory:

myprog/db.c -- source for the db module
myprog/db.h -- header for the db module
myprog/db_test.c -- unit tests for the db module
myprog/testdata/file1.db -- sample database for testing
myprog/myprog.c -- main module
myprog/myprog_test.c -- integration tests for the whole program
myprog/utils.c -- source for the utils module
myprog/utils.h -- header for the utils module
myprog/utils_test.c -- unit tests for the utils module

How
***

The changes proposed above should, and have to be, done iteratively.

It is "easy" to pick a subset of tests from src/tests and move them
around the source tree, doing all necessary file list adjustments at
the same time.  This will minimize breakage, huge changes to the tree
all at once and make the modifications to the filelists bearable.

Smaller changes also give us a chance to "canary" the changes in a few
subdirectories and evaluate how well things look.  The easiest place
to start, I think, is src/tests/util/.

Comments?

-- 
Julio Merino