Re: Running tests with different path for Kyuafiles and binaries ?

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Fri, Jun 8, 2012 at 6:25 AM, Emmanuel Fleury <fleury%labri.fr@localhost> 
wrote:
> Hi,

Hello!

> I probably have missed something, but I have the following problem.
>
> I am using ATF+Kyua in a project that I am running. Some of the developers
> working in the project prefer to keep the sources away from the built files
> and, so, create a 'build/' directory and execute the configure file from
> this location.
>
> The result for the test framework is that the test binaries will lie in the
> $(top_builddir) directory and the Kyuafile will stay in the $(top_srcdir)
> directory.

Unfortunately, this is trickier than it should be.  You can take a
look at how the kyua-cli package jumps through the necessary hooks to
get this working... but it's not pretty.  Basically, there is a script
to populate the build tree with copies of the Kyuafiles, just as if
they were products of the build.  It works, but it's extra complex.
Ideas on how to simplify this would certainly be welcome.

> I am already running 'kyua test --kyuafile=$(top_srcdir)' from my Makefile
> in order to find the Kyuafiles. But, it breaks when I am specifying:
> atf_test_program{name="$(top_builddir)/my_test"}

At the moment, the *_test_program definitions restrict the test
programs to live in the same directory as the Kyuafile that references
them; passing paths will not work.  I'm not convinced this is the most
reasonable solution, but I like its simplicity.

Having Kyuafiles pointing at arbitrary directories could make things
more confusing.  For example, how do you identify the test programs
within that Kyuafile?  By their basename?  Or should there be an
additional "binary" attribute to the test program that can be a path,
and "name" is just the identifier?  How are these represented for
historical purposes in the database?

Another possibility that just crossed my mind (and that I think makes
a lot of sense atually) would be to teach "kyua test" about build
directories: allow the caller to specify a destdir/buildroot and use
that as the prefix to locate the binaries themselves (not the
Kyuafiles).  This would be more in line with make's behavior and some
other build tools I have used.  The issue here would be dealing with
programs that reference data files in the program's srcdir: if the
data file needs some kind of "build", then it will be in the
buildroot; otherwise, it will be in the source tree.  But we could
deal with that somehow!

What do you think?

> Very likely because the '$(top_builddir)' is not replaced by its value.
>
> I also tried to play with configuration variable and to add a 'top_builddir'
> variable. But, it seems that the variables are just fixed for once and no
> extra variable can be added through the configuration file.

Configuration variables are not exposed to Kyuafiles.  I guess it
would make sense to do so though.

-- 
Julio Merino / @jmmv