On 05/11/2007, at 13:49, Hubert Feyrer wrote:

> On Sun, 4 Nov 2007, Julio M. Merino Vidal wrote:
>> Please raise your concerns quickly.  I probably won't be able to  
>> do all this for the following two weeks, but who knows, maybe I'll  
>> find time ;-)
>
> No concerns, but: what do we do with it?

First of all, I think it's an excellent tool for developers.  Every  
time you touch a specific piece of code, say sh(1), you can easily go  
to the tests tree, run the tests and ensure that you have not broken  
anything.  I have found this procedure an invaluable tool when  
working on projects that do have extensive test suites that can  
easily be executed at will.

Then, it'll also be useful to end users.  For example, after you have  
successfully installed a release on a production machine, you can  
easily see that it passes some basic tests to ensure it 1) remains  
stable and 2) behaves as expected.  At the moment most people just  
"builds a release" as a stress test, but the idea is that you can  
have something more accurate and powerful.  This is specially  
interesting on platforms that see less testing; we all know that  
there is much more (obvious) breakage in them than, say, i386.

As a matter of fact, there are a couple of sh(1) regression tests  
that fail at the moment in current, and I bet they remain unfixed  
because no one actually executed the test suite to discover them  
(which is understandable because it's not trivial to do).  Some other  
tests (the ones for df(1)) pass in i386 but fail in amd64.  I expect  
people will have more pressure in fixing those problems if they can  
be notified of them early enough (or notified at all).

> How does this tie into the NetBSD release process? Will it be ran  
> as part of the daily builds, or is there a dedicated machine that  
> runs this, and posts reports? While I understand that this is  
> technically better than src/regress, how do we solve the problem of  
> actually getting those tests ran on a regular base?

There are no plans on this.  To get started, I'd say "don't branch/ 
release if any test fails".  That'd be something for releng to do,  
but I'd really like to see some machine running the test suites  
periodically, and on as many platforms as possible.  (That's  
basically the point of HTML reports, so that we'd collect them into a  
single machine and easily show them to interestd developers.)

We can sort this out later, once people has experimented with the new  
framework, raised concerns about it, and when we have more tests in  
the suite.

> Oh, and from the 'documentation' department: Is there some "intro"  
> text that shows a software author how to write tests for his software?

I added some example to the web site, but I haven't got the time to  
write documentation.  (There are manual pages for all the tools,  
though.)  Even though, writing tests is very easy I think, so using  
others as an example should not be too problematic for now.  Lame  
excuse for lack of formal documentation, I know, and I'll eventually  
get to writing some.

> (And I'm not going to ask if we're going to make regression tests  
> mandatory for all code that we import in the future now :-)

We should! :-)  But again, this is a policy that can be enforced  
later on when people gets used to the framework and sees the value of  
it.

Cheers,

-- 
Julio M. Merino Vidal <jmerino@ac.upc.edu>