Re: CVS commit: src/tests/syscall

[Julio Merino <jmmv%julipedia.org@localhost>]

On 05/21/11 21:58, Christos Zoulas wrote:
> On May 21,  9:26pm, jmmv%julipedia.org@localhost (Julio Merino) wrote:
> -- Subject: Re: CVS commit: src/tests/syscall
> 
> | Just to make sure we are talking about the same: you were running the
> | test program with atf-run; correct?  If yes, what platform was this in?
> |  (I'm testing with amd64.)  Otherwise, this is expected behavior: tests
> | are not supposed to be executed without atf-run.
> 
> No, I am running it directly. Programs could warn if they figured out
> that they are not running under atf-run. Print something like: not running
> under atf-run, timeouts will not work.

Sounds like a good idea.  Note that there is *many* more things that
will not work.  The test case will dump garbage in the current directory
and it won't be cleaned, the environment won't be sanitized, child
processes won't be killed...

> Also it would be nice if programs
> when invoked with -l or something they would list all their tests, and if

That already works :-)  The output format is oooogly though

> invoked with -a they would run all the tests.

Tests used to do exactly that more than a year ago, but it turned out to
be a bad idea...

The test programs were extremely complex: they had to implement all the
test case isolation (work directory, timeouts, etc.) themselves, and
that meant having to implement the same thing in C, C++ and shell.

Also, and this is what bugged most people: debugging the test cases was
almost impossible due to all the black magic that was involved in them.
 The test cases would delete temporary files on failure and gdb was
unusable.

As a result, I moved all the isolation logic into atf-run and turned the
test programs into a 'dumb' frontend for the test cases.  Test programs
now don't do much more than just running test case bodies and exposing a
consistent interface, so they really should not be used directly unless
you need to debug a particular test case.  Oh, and the whole system
works much, much faster ;-P