Format of test programs' output

["Julio M. Merino Vidal" <jmmv84%gmail.com@localhost>]

Hi!

I have started cleaning up the current mess of data formats and their  
parsing.  I would like to get some data formats that can remain  
stable for several versions and be easy to process in multiple  
situations.  The former is, IMO, a pre-requisite for integrating ATF  
into NetBSD; otherwise frequent changes in the format will cause pain  
to users even if the differences are handled gracefully.

So I have started with the format that test programs print during  
execution.  Remember that, at the moment, a test program is just a  
collection of test cases that presents a consistent CLI to the user.   
There are requisites for this format:
- As these are programs, we cannot restrict them from printing stuff
  to their stdout and stderr, hence we must be able to capture these
  two as part of the test program's execution and not be disturbed by
  data that should not be there (e.g. printed by the framework itself).
- Test cases can take a while to execute, yet we want to keep the
  user informed of progress.  We must print some stanzas before the
  test case is executed and after this happens.
- The results are something separate from stdout and stderr.  The
  reader should be able to parse this alone and not be disturbed with
  other output from the program.  In other words, the results cannot
  be printed to the stdout.

Here is the format I have in my mind:

tcs-count: <number of test cases that follow>
tc-start: <name of 1st test case>
tc-end: <name of 1st test case>, passed|failed|skipped[, reason]
tc-start: <name of 2nd test case>
tc-end: <name of 2nd test case>, passed|failed|skipped[, reason]
...
tc-start: <name of nth test case>
tc-end: <name of nth test case>, passed|failed|skipped[, reason]
<end of file>

Some remarks:
- Reason is only present for failed and skipped test cases.
- The test case name is repeated on the tc-end line so that it is easy
  to grep for specific test case results, without having to filter it
  through atf-run nor atf-report.  This can be convenient in multiple
  situations.  Simply doing something like "grep '^tc-end: foo, passed'
  should be enough to ensure that the test case 'foo' passed, for
  example.  This is not possible with the current format, which prints
  stuff on different lines.
- The total number of test cases is printed at the beginning so that
  the reader can know how many test cases are left during execution
  and provide better status messages to the user.

As regards stdout and stderr, it is important to separate the output  
generated by each test case (if any) for reporting purposes.  E.g.  
when a test case fails, we want to show all its output to the user  
(read, developer) for diagnosis purposes.  I haven't found any way to  
synchronize different streams with each other... so I force test  
programs to print "__atf_tcs_separator__" on a line of its own when  
they have finished.

This is, of course, all handled by the ATF libraries so test cases  
don't have to implement any of the above on their own.

Do you see any important data/requirement missing from the above?

Thank you,

--
Julio M. Merino Vidal <jmmv84%gmail.com@localhost>