Re: test case metadata

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Apr 19, 2010 at 10:07 AM, Martin Husemann 
<martin%duskware.de@localhost> wrote:
> On Mon, Apr 19, 2010 at 10:00:33AM +0100, Julio Merino wrote:
>> * And second, to allow the user to query such metadata.  This was
>> supposed to let end users run test cases based on a query to such
>> metadata.  For example, run all tests related to bug number X, or run
>> all tests that require root because you just ran sudo.  At the moment,
>> none of this is implemented, and I'm not sure if it is valuable at all
>> to keep.
>
> I still think this is valuable to have,

Agreed, but I think we should simplify the declaration of tests.  In
most situations, the description feels redundant, and having to add
the boilerplate code to define an empty head makes the test programs
code too verbose.  (I'm just comparing our code to, for example,
py-unit tests, which result in a much more implementation.)  As long
as we can simplify this, I'm happy to keep it.

> and one important meta-data item
> should be added: expected test run length (realy big thumb figure is
> ok, either absolute or if cpu bound per cpu/mhz or something).

What I've seen in some other test suites is a size tag, say "small" or
"large", or a type tag, say "unit test" or "integration test" (with
the idea that unit tests must be very quick to run).  Would this suit
your idea?

> Having a way to run all quick tests with a limit of expected time per test
> as a command line option and maybe a summary of total expected time upfront
> would be very usefull.

Indeed.  I really would like to finish fixing a couple of regressions
in the nofork branch and merging it into mainline.  Implementing this
with the new code base would be pretty easy (atf-run reads the
metadata of all test programs, so filtering them is trivial).

-- 
Julio Merino