Re: About non-deterministic tests

[Julio Merino <jmmv%NetBSD.org@localhost>]

On 9/18/11 9:11 AM, Jukka Ruohonen wrote:
> Hi.
> 
> This is not really related to ATF, but more of a question related to testing
> generally and specifically testing done in NetBSD.
> 
> In NetBSD, and I guess in anything that does system testing, many tests for
> known defects are not failing always but just occasionally. The examples
> range from race condition tests to different timing tests. However, it
> would be much easier to keep track of the results if the baseline would be
> "zero failures" (at least on the architectures that are part of the daily
> schedule).
> 
> Now perhaps it would be possible to modify these tests so that they always
> fail. But as we are reaching 3000 test cases, it does not seem reasonable
> that few tests are trying to catch failures by running ten minutes. In
> addition, it is known that Qemu contains severe timer issues (cf. Anita).

Yeah, this is a feature requested by pooka@ a long time ago and it would
definitely make some tests easier to write.

I think we should have a way to:

- Declare this kind of test cases more easily, without having to
  resort to weird expect/fail combinations.
- The user should be able to tune for how long the race condition
  tests run.  The defaults should probably be to run for a reasonable
  amount of time (10 minutes is not reasonable), but the user may
  want to run things in a more exhaustive manner.  Plus, with
  parallel execution, running long tests should not be that of a big
  deal.

I have filed http://code.google.com/p/kyua/issues/detail?id=27 with the
details above.  Do you think this covers what you have in mind?

-- 
Julio Merino / @jmmv