Re: About non-deterministic tests

[Julio Merino <jmmv%NetBSD.org@localhost>]

On 9/19/11 12:26 AM, Jukka Ruohonen wrote:
> On Sun, Sep 18, 2011 at 02:43:32PM -0400, Julio Merino wrote:
>> - Declare this kind of test cases more easily, without having to
>>   resort to weird expect/fail combinations.
> 
> An "unexpected failure" as noted in the bug report sounds fine. Another
> option would be to use atf_tc_skip() at the end if the failure does not
> trigger.
> 
>> - The user should be able to tune for how long the race condition
>>   tests run.  The defaults should probably be to run for a reasonable
>>   amount of time (10 minutes is not reasonable), but the user may
>>   want to run things in a more exhaustive manner.  Plus, with
>>   parallel execution, running long tests should not be that of a big
>>   deal.
> 
> Perhaps this is not entirely satisfactory solution generally. There is still
> the problem inherited in the NetBSD's testing infrastructure; the timers in
> Qemu are not reliable, so any "race condition test" is bound to be unreliable.
> Increasing the timeout values may not help in all cases.

Well, but "race condition tests" are unreliable by definition.  You
cannot expect the race condition to always trigger no matter for how
long they run.

Also, what I meant by "how long" was a measure in "number of iterations"
rather "amount of time".  Both make sense though.

> The same applies to real hardware too. While it may be entirely reasonable
> to test whether "x" is in nanosecond scale on some architecture and/or
> hardware configuration, it is hard, if not impossible, to write an
> universal/portable test case for something like this.
> 
>> I have filed http://code.google.com/p/kyua/issues/detail?id=27 with the
>> details above.  Do you think this covers what you have in mind?
> 
> Sounds good.

Aha!  I knew I had written down some similar notes somewhere about
exactly this.  The TODO file in the ATF distribution (it's not in the
NetBSD tree though) contains some more ideas about this.  You can access
these online here:

http://mtn-host.prjek.net/viewmtn/atf/revision/file/7ae5fa10200ec66c26f08c27e4aeaf3facf3f031/TODO

... and scroll down to the "Implement race-condition tests" section.

-- 
Julio Merino / @jmmv