About non-deterministic tests

[Jukka Ruohonen <jruohonen%iki.fi@localhost>]

Hi.

This is not really related to ATF, but more of a question related to testing
generally and specifically testing done in NetBSD.

In NetBSD, and I guess in anything that does system testing, many tests for
known defects are not failing always but just occasionally. The examples
range from race condition tests to different timing tests. However, it
would be much easier to keep track of the results if the baseline would be
"zero failures" (at least on the architectures that are part of the daily
schedule).

Now perhaps it would be possible to modify these tests so that they always
fail. But as we are reaching 3000 test cases, it does not seem reasonable
that few tests are trying to catch failures by running ten minutes. In
addition, it is known that Qemu contains severe timer issues (cf. Anita).

So maybe something like

                if (failure) {
                        atf_tc_expect_fail("PR xxx/123");
                        atf_tc_fail("expected race occured");
                }

                ...

                atf_tc_fail("expected race did not occur");
                // end of test

Or should there be a new category in addition to "failure" and "expected
failure". Other ideas?

- Jukka.