Re: xfail: expected failures

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Mon Jun 28 2010 at 12:31:33 +0100, Julio Merino wrote:
> On Mon, Jun 28, 2010 at 12:16 PM, Antti Kantee <pooka%cs.hut.fi@localhost> 
> wrote:
> > On Mon Jun 28 2010 at 07:47:56 +0100, Julio Merino wrote:
> >> On Mon, Jun 28, 2010 at 2:09 AM, Garrett Cooper 
> >> <yanegomi%gmail.com@localhost> wrote:
> >> >
> >> > BTW, this comment from the wiki looks wrong:
> >> >
> >> >   atf_tc_expect_pass();  /* If reached, immediately exits with failure. 
> >> > */
> >>
> >> Well, the comment is actually right.  It is a matter of
> >> simplification: if the test case was expecting any kind of fatal
> >> failure, attempting to change the expectations is an immediate error
> >> because that fatal failure did not happen.  It'd be possible to do
> >> some other thing, but it'd be too complicated and require active
> >> communication between the test case and atf-run.
> >
> > The comment will become wrong once the feature under test is fixed and
> > the failure expectency is removed.  However, your test will do nothing
> > to draw attention to this.
> 
> That's not the case.  We have the following:
> 
> atf_tc_expect_death("The following code will crash");
> abort() / exit()
> atf_tc_expect_pass();

I did write "comment", so it's a little unfair to analyze a situation
in which you remove the comment... ;)

> > Also, another thing that came to mind is that the semantics don't allow
> > to write tests which can fail non-deterministically in two places:
> >
> >  stage1();      /* might fail here */
> >  prep_stage2(); /* should not fail here and must be done after stage 1 */
> >  stage2();      /* should latest fail here */
> >
> > I don't know how common such a scenario would be, but simple gating with
> > xfail_enter() / xfail_exit() works there too.
> 
> At the moment, every xfail enter/exit block must have at least one
> failure in between.  It's more of an "all block must fail" than just a
> "at least one block should fail".  Not sure which of the two
> approaches makes more sense.  If we stick to the principle of "each
> test case must test one scenario", then expecting a failure on just
> one block makes sense and expecting a failure in every block probably
> does not.  I think.

How many times can a test exit or timeout or get a fatal signal?

> If you have two conditions that might fail, just put them in the same
> block as non-fatal checks.

You can't if the condition is a crash.

> How realistic is that you need to do other
> stuff in between for which you must do regular checks?

It's more a question of being a prep stage which you do not except
to fail.

Anyway, my main gripe is still with the totally misleading name.
But whatever, I'll live ;)