Re: xfail: expected failures

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Wed, Jun 16, 2010 at 9:49 PM, Antti Kantee <pooka%cs.hut.fi@localhost> wrote:
> [Julio requested I send a small note to this list since there have been
> others interested in the feature]

Thanks for the work and sending the email!

[...]
> Xfail will be present in atf 0.10 (according to Julio ;).  The current
> implementation is a big binary hammer, meaning it is not possible to
> specify where the test is expected to fail.  This may change for the
> 0.10 release.  If anyone has any other ideas about what they would like to
> see in the this department, I guess now would be a good time to speak up.

I'm planning on writing a small design proposal with the use cases we
discussed privately before implementing this feature in mainline.
It's a bit tricky and I would like to get it "right" from the start.
Expect something tomorrow or friday :-P

So far, the use cases are:
- Catch crashes as expected failures.
- Do not detect preparation failures as expected failures; only some
assertions/crashes are failures.
- Probably have ATF_CHECK* macros to check for expected failures, as
in some cases these are enough.
- atf-run can implement this feature but test programs themselves
should also do it.

Any early comments?

-- 
Julio Merino