Re: test case metadata

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Apr 19, 2010 at 10:22 AM, Antti Kantee <pooka%cs.hut.fi@localhost> 
wrote:
> On Mon Apr 19 2010 at 11:07:53 +0200, Martin Husemann wrote:
>> On Mon, Apr 19, 2010 at 10:00:33AM +0100, Julio Merino wrote:
>> > * And second, to allow the user to query such metadata.  This was
>> > supposed to let end users run test cases based on a query to such
>> > metadata.  For example, run all tests related to bug number X, or run
>> > all tests that require root because you just ran sudo.  At the moment,
>> > none of this is implemented, and I'm not sure if it is valuable at all
>> > to keep.
>>
>> I still think this is valuable to have, and one important meta-data item
>> should be added: expected test run length (realy big thumb figure is
>> ok, either absolute or if cpu bound per cpu/mhz or something).
>>
>> Having a way to run all quick tests with a limit of expected time per test
>> as a command line option and maybe a summary of total expected time upfront
>> would be very usefull.
>
> I'd just make it simply fast/slow.  I don't think we have much which
> would last anywhere between 1s and 10000s, usually below and over.

Correct.

> However, there are tests which pass if they don't crash in a certain unit
> of time.  Examples are the fs rename race tests.  They are currently
> hardcoded to last 10s.  Being able to configure the duration on the
> command line might be useful.

You should be able to do that now.  You can pass arbitrary variables
to the test through the command line, in one of the following forms:

./t_rename -v deadline=300 foo-test-case
or
atf-run -v deadline=300 t_rename

You can then query this deadline variable using atf_tc_get_config_var
in C or atf_config_get in sh.  Just be sure to set a default value for
them so that they are not required.

-- 
Julio Merino