Hello,

[ This is something I outlined in a previous, unrelated mail, so I'm  
reposting it now in its own thread with more details. ]

[ CC'ing my project mentor ]

One of the "key features" of atf is the ability to install test  
programs into the target system (i.e. under /usr/tests/...) as a  
natural step so that end users can easily run them all.  This makes  
me think what should the structure of those installed tests be, which  
is "of course" (not necessarily, but conveniently) correlated to the  
structure of the test programs in the source tree.

The first possibility is to organize test programs as is done now in  
src/regress: that is, mimicking the organization of the source tree.   
This way, we'd have something like the following:

     /usr/tests/bin - Tests for programs built from src/bin.
     /usr/tests/usr.bin - Tests for programs built from src/usr.bin.
     /usr/tests/lib - Tests for libraries (/lib and /usr/lib).
     /usr/tests/share/mk - Tests for mk makefiles.
     /usr/tests/sys/kern - Tests for core kernel functionality.
     /usr/tests/sys/fs/{ntfs,udf,tmpfs,...} - Tests for kernel file- 
systems.
     Etc.

The advantage of this approach is that, given a specific *file* in  
the system, you can guess where its tests are.  It suits quite well  
tests for basic tools such as /bin/ls, /bin/cp, etc. and also some  
subsystems (such as the kernel) because they are already self- 
contained in the source tree.

The downsides are that logically-related tests are completely  
spread.  A couple of examples:

- If I wanted to test make and all of its infrastructure, I'd like
   to go to a single directory and run all the tests.  This could
   ensure that make(1) works and that all the bsd.*.mk files behave
   correctly.
- If I wanted to test the atf framework, what'd I do?  The tests for
   its tools could be located in tests/bin, tests/libexec... the tests
   for its libraries in tests/lib/libatf, the tests for its shell
   routines in tests/share/atf, etc.  It'd be painful to run them all,
   and I couldn't know which is the correct "order" (from most basic
   unit testing to more complete integration tests).

This could be somewhat solved by tagging each test program with the  
logical application it belongs to.  But still, this doesn't preserve  
the order, and running all the tests for a specific component could  
involve scanning the full tests tree, which I bet will be very  
large.  (This will most-likely be implemented, but shouldn't be  
relied on for regular operation given how slow it will possibly be.)

The opposite approach is to organize tests logically both in the  
source tree and in the installed files.  And instead of inventing a  
brand-new organization, it has just occurred to me that we could  
follow the structure of syspkgs (I know, they are not here yet, but  
the components are already defined).  So, considering that all  
packages follow the following scheme: set-component-type (for lack of  
better names), I'd pick the component part (the second element) and  
use that as a directory name under /usr/tests.  Then, put all related  
tests into there.

For example, skimming through the list of current sets, we'd have:

     /usr/tests/amd/...
     /usr/tests/atf/...
     /usr/tests/bind/...
     /usr/tests/calendar/...
     /usr/tests/games/...
     /usr/tests/util/...
     Etc.

Each component could be free to organize tests under its hierarchy as  
it wanted.  The most obvious example are the tests for the kernel,  
which can easily be categorized into different areas.

The advantage of this approach is that the user could quickly find  
all the tests related to a given logical *component*.  And this can  
make a lot of sense if we get syspkgs some day.

The disadvantage is that the source tree for the tests does not  
follow the source tree for the code, which can cause some confusion  
among developers.  (A possible fix for this could be to put the  
source code for the tests *alongside* the source code for the  
programs/auxiliary files/whatever.  But that can make things complex.)

At the moment I prefer the second option basically because when  
someone is editing a system component, he wants to easily run all  
logically related tests.  It doesn't make sense to be editing the  
pkg_install's code and have to run the tests for postinstall, just  
because the two happen to be under the same usr.sbin source  
directory.  Are there strong objections against it?

Kind regards,

-- 
Julio M. Merino Vidal <jmmv84@gmail.com>