Re: Reorganizing src/tests

[Julio Merino <jmmv84%gmail.com@localhost>]

On Thu, Jul 15, 2010 at 2:17 PM, Mindaugas Rasiukevicius
<rmind%netbsd.org@localhost> wrote:
[...]
> However, what would you do for kernel tests?

As I said, tests that are not easy to classify can go/remain under
src/tests/.  I'm not concerned about what to do with such tests at
this point -- we can decide later separately.  Should have stated this
clearly.

>> 1) If we have this source layout and want to match the installed
>> layout, then the tests have to get installed under
>> /usr/tests/bin/cp/tests/, which is silly.  If we put the tests in
>> src/bin/cp/tests/ and install them in /usr/tests/bin/cp/, then we are
>> losing consistency which I think is a bad thing.  (Note that tests can
>> query what their "source directory" is, and such inconsistency will
>> cause lots of confusion.)
>
> Why?  Convention is consistent:
>
>        <subdir>/<app> and /usr/tests/<subdir>/<app>
>
> For <app>/tests/ directory - grouping and being near the source are
> the points, not much of "(in)consistency" there.

What do you mean?  What I proposed is exactly this: install the tests
for subdir/app/ in /usr/tests/subdir/app/.  So, effectively,
/usr/tests/ ends up having the same layout of src/.

But if you add the tests/ subdirectory to the *sources*, then
subdir/app/tests/ must still get installed under
/usr/tests/subdir/app/ (because the extra tests/ component here would
be pointless), and this is not consistent.  Sure it can be made to
work, but I smell confusion.

>> 2) By introducing the tests/ subdirectory under each other directory,
>> we are still hiding tests.  E.g. if I'm editing cp.c and I see right
>> next to it a file called cp_test.c, I'll be tempted to edit it.  If
>> cp_test.c is under tests/cp_test.c, I can easily miss it.  Note that
>> we don't put manual pages in a man/ subdirectory, and for a good
>> reason.  Oh, and directories in cvs are expensive (braindead?).
>
> Code should be convenient to work with.  Hence, well organized, taking
> into account directory level too.  Messing up all parts of application
> into a single directory would be going to the opposite direction.

In most cases, the subdirectories will get in the way.  Sure, for your
libpthread example it can make sense to split things up.  But for
virtually all other source directories (bin, sbin, usr.bin, usr.sbin
in particular) they provide no value because each application
directory contains very few files.  And anyway: does it really make
sense to split things by *file type*?  Wouldn't it make more sense to
split them by logical concept?  E.g. a subdir for mutexes code, a
subdir for conditions code, etc.

(Regarding "code should be convenient to work with", I've heard the
exact opposite argument as the one you propose: some projects prefer
to dump all the code in the root of their project because then it's
just right there, easily available with autocompletion.  So I suspect
this is a matter of personal taste.)

The thing is I recently killed the tests/ tree from the atf package
and intermixed the tests with the source code (following the approach
I proposed here).  And oh boy, it's a joy to work with them now!

One more thing: most of these tests/ subdirectories will end up
containing exactly two files: a trivial Makefile and the test program
itself.  If we avoid the subdirectory altogether, we can cut down on
an extra Makefile + make recursion because we just need *one* extra
line in the program's Makefile to get the tests working.

We probably shouldn't have a strict policy as to how to organize the
tests: if for a particular subdirectory it makes sense to create a
tests/ subdir, you can do so, but I'd like the recommended policy to
be to not create such directories.

[...]

>> It is "easy" to pick a subset of tests from src/tests and move them
>> around the source tree, doing all necessary file list adjustments at
>> the same time.  This will minimize breakage, huge changes to the tree
>> all at once and make the modifications to the filelists bearable.
>
> Perhaps adding "make test" facility to each Makefile would be useful?

Already exists :-)  (But for now it only works from subdirectories of
src/tests/.  If we move this proposal forward, it will work right from
where the program sources are.)

-- 
Julio Merino