On Wed, 22 Mar 2006 05:59:36 -0800
Ben Collver <collver@peak.org> wrote:
> I get the feeling that regression testing is used to insure that certain
> "dumb bugs" don't appear.
> 
> I skimmed a book titled The Craft Of Software Testing: Subsystem Testing
> by Brian Marick.  It was interesting, but setting up the tests seemed
> like a lot of work.  I could see using it for a device driver or some
> small component.  But for a system like pkgsrc or NetBSD, I would need
> the computer to help out more.

What you want is "unit testing."  The idea is to have a suite of test scripts that runs from a single command (ideally from a cron job daily) that spits out problems.  In fact, the proper way to do this is "test first."  I have started doing this in PygreSQL (http://www.PyGreSQL.org/) to prevent regressions.  For example, I recently found that some code got changed that introduced a bug.  Before fixing the bug I modified the unit tests to prove that there was a problem.  I then fixed the bug and rant the unit tests again to prove that the bug was fixed.  Note that the test has to fail first otherwise you don't know if it is a valid test.

Unit testing theory doesn't say that the entire system has to be tested.  Write as many tests as you can, adding from time to time, but make a rule that nothing ever gets changed again unless a unit test is made first.  The idea is that you can't possibly catch every possible bug in advance but with a little discipline you can make sure that you never have to fix the same bug twice.

I have a few major projects that I run that have unit testing as an integral part of the development process and those programs run the unit tests from a cron job daily and mail any error output to all of the programmers and project leader.  This makes sure that no bug goes unnoticed for more than 24 hours.  It is much easier to fix those than to try and figure it out six months later.

I use the Python unit testing framework.  See http://pyunit.sourceforge.net/ for more information on this framework.  While primarily designed for testing Python programs, it can be used to test anything that it can run.

-- 
D'Arcy J.M. Cain <darcy@NetBSD.org>
http://www.NetBSD.org/