Re: Importing Kyua into NetBSD

[Julio Merino <julio%meroh.net@localhost>]

On Sat, Sep 1, 2012 at 6:49 PM, Jean-Yves Migeon
<jeanyves.migeon%free.fr@localhost> wrote:
>
> FWIW the steps look fine to me, however:
>
> 1) I would provide the two knobs (MKATF and MKKYUA) for builds:
> - saves up build time if required, and allow anyone to build only on or the
> other
> - make the transition a bit smoother if you want to deprecate ATF. Having
> KYUA protected with a MKATF knob is somewhat disturbing (at least to me :) )

It's weird, yes, but here is my rationale for using MKATF: today, all
of our tests rely on the ATF libraries (which are not going away).
Because Kyua is supposed to be used with these tests, if you set
MKATF=no, the build of Kyua becomes, in turn, useless.  I'd argue that
MKATF should be called MKTESTS or similar, so you get an all or
nothing and you avoid relying on the name of the tools.

But if people prefer a MKKYUA (Paul said the same thing), I'm fine
with that too.  We can revisit the naming scheme later.

> 2) Speaking about the "user validation period": what about tracking
> regressions between an ATF run and a KYUA run and comparing results?

That's an interesting detail that I had missed!  I've been doing this
manually since Kyua had support for all the features needed by the
NetBSD test suite... but something more automated would be great for
this transitional period.

> ATF uses XML coupled with xsltproc to generate an HTML report, while kyua
> outputs HTML directly. What would be the best way to validate that no
> regression happened after an adjustment? Comparing HTML outputs "visually"?
> Parsing the ATF db and KYUA sqlite db manually? Something else?

I'll try to think about something in more detail and update the
document, but a simple idea could be: because we should always have a
passing test suite, you'd compare the exit code of atf-run and "kyua
test" and rely on that at least for the most broad sense of passing /
failing.  anita could be taught to run both in sequence and compare
the results.

Thanks for the review!

-- 
Julio Merino / @jmmv