Re: Revisiting the migration path to Kyua

[Julio Merino <jmmv%outlook.com@localhost>]

On Feb 16, 2014, at 7:13, Andreas Gustafsson <gson%gson.org@localhost> wrote:

> Julio Merino wrote:
>> If we go with this proposal, we need to change Anita to decouple the
>> enabling of Kyua in the base system by default with the selection of
>> the tool used in your test machines.
>> 
>> I think it'd make sense to add a command-line flag to "anita test"
>> to let the caller specify whether atf or kyua is to be used -- and
>> to ignore MKKYUA altogether.
> 
> Would we not also want the ability to run both?  Will there be a
> transition period when some tests are only in ATF and others are
> only in Kyua?

No, I'd rather not do that because maintaining two different test sets will 
become ugly very very quickly.  Both tools should run the exact same tests.

> Also, the command line flag approach is awkward for testing historical
> versions, as you would have to manually figure out the appropriate
> setting for the particular version you want to test.  It might be
> better for anita to try running both ATF and Kyua, and for the
> higher-level test harnesses that generate HTML reports to publish the
> results from whichever one worked, or both.

Maybe, but as you say there are problems that prevent you from doing that right 
now.  Also, doing that would double the time required to run tests, and we know 
that qemu is super-slow for this... so I'm not sure it's a good idea.

We could reassess this later though because running the tests through both 
tools would provide a mechanism to compare results and ensure that they match.

The idea of the flag is only so that developers (I suspect only me though) 
using anita can choose to test the new tool iff they want to.  Your test 
machine should never have to use that flag: it should use the default behavior 
in anita and, when we decide that kyua is good enough, flip the default which 
would then switch the behavior of your machines.

>> I think the changes should be easy.  Should I try to come up with a
>> diff?
> 
> I can make the changes to anita once we have decided what they
> should be.
> 
> But I'm afraid I still don't quite understand why this change in
> transition strategy is needed.  The main problem I found with
> turning on MKKYUA in the current setup was a dramatic increase in
> disk space usage for the test results, and I don't think this change
> will solve that problem

Right, it won't, and the whole point of this change is to hide the problem from 
you so that progress can be made _on other fronts_.  (Don't get me wrong: I do 
have plans to address your concerns, but they involve quite a bit of work and I 
haven't been able to commit the necessary time.)

I'm hoping that once people have an easier way to start testing kyua they will, 
and I'm sure that _more_ problems will be discovered once that happens.  I'd 
rather uncover what these issues are sooner rather than later.

> but only postpone the growth until the
> existing tests are migrated, so what problems will it solve?

With "tests" you mean "machines running tests" here?