Re: test case metadata

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Mon Apr 19 2010 at 13:05:10 +0100, Julio Merino wrote:
> > On Mon Apr 19 2010 at 10:45:41 +0100, Julio Merino wrote:
> >> You should be able to do that now.  You can pass arbitrary variables
> >> to the test through the command line, in one of the following forms:
> >>
> >> ./t_rename -v deadline=300 foo-test-case
> >> or
> >> atf-run -v deadline=300 t_rename
> >>
> >> You can then query this deadline variable using atf_tc_get_config_var
> >> in C or atf_config_get in sh.  Just be sure to set a default value for
> >> them so that they are not required.
> >
> > The good thing about arbitrary values is that everyone can have their
> > own ;)
> >
> > If we want something really supported by the test framework, it should
> > be enforced by the test framework instead of letting each testcase do
> > a little of their own magic.
> 
> Yep, true.  So... you have a test case that runs in a tight loop
> forever, and a property that says: if the test has run successfully
> for more than X seconds, stop it and mark it as passed.  Correct?
> Could you file a PR and assign it to me with this request? :-)

Well, yes and no.  I think it's still useful for the test to verify it
can complete.  Otherwise it might have hit an undetected deadlock and
we falsely report the test passed.

What we could do is pass in test_length and fail any test which uses
that value and doesn't complete in test_length+2 seconds (or +3 or
+5 or whatever stetson-harrison value is reasonable.  let's solve the
halting problem some other day).  Alternatively, we could fail all fast
tests which don't complete in the grace time (maybe if the test sets
``timeout'', we need to take it into account and wait until timeout to
fail the test).

opinions?