Re: About non-deterministic tests

[Jukka Ruohonen <jruohonen%iki.fi@localhost>]

On Sun, Sep 18, 2011 at 02:43:32PM -0400, Julio Merino wrote:
> - Declare this kind of test cases more easily, without having to
>   resort to weird expect/fail combinations.

An "unexpected failure" as noted in the bug report sounds fine. Another
option would be to use atf_tc_skip() at the end if the failure does not
trigger.

> - The user should be able to tune for how long the race condition
>   tests run.  The defaults should probably be to run for a reasonable
>   amount of time (10 minutes is not reasonable), but the user may
>   want to run things in a more exhaustive manner.  Plus, with
>   parallel execution, running long tests should not be that of a big
>   deal.

Perhaps this is not entirely satisfactory solution generally. There is still
the problem inherited in the NetBSD's testing infrastructure; the timers in
Qemu are not reliable, so any "race condition test" is bound to be unreliable.
Increasing the timeout values may not help in all cases.

The same applies to real hardware too. While it may be entirely reasonable
to test whether "x" is in nanosecond scale on some architecture and/or
hardware configuration, it is hard, if not impossible, to write an
universal/portable test case for something like this.

> I have filed http://code.google.com/p/kyua/issues/detail?id=27 with the
> details above.  Do you think this covers what you have in mind?

Sounds good.

- Jukka.