>Number:         23952
>Category:       toolchain
>Synopsis:       "make regress" lacks critical infrastructure
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    toolchain-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Fri Jan 02 18:18:00 UTC 2004
>Closed-Date:
>Last-Modified:
>Originator:     Martin Husemann
>Release:        NetBSD 1.6ZG
>Organization:
>Environment:
System: NetBSD night-porter.duskware.de 1.6ZG NetBSD 1.6ZG (PORTER) #0: Thu Jan 1 14:05:24 CET 2004 martin@night-porter.duskware.de:/usr/src/sys/arch/i386/compile/PORTER i386
Architecture: i386
Machine: i386
>Description:

Regression tests are important. We have "make regress" for this, and it partly
works. Just to stress the point how important this is: when I started trying
this target last week, I could imediately panic the kernel on two archs
(i386 and sparc64) by running programms in the regression suite.

The fact that this problems have been so easily detectable (by just running
"make regress") and been undiscovered for a non trivial time shows there
is something wrong.

After playing with the regression tests now for some time, I found our 
infrarstructure for it lacking in several ways. Here are the most critical
ones I see, please append to the PR if you know others.

 (1) There is no (optional?) automatic logging and summary at completition
     This is a SMOP, should be easy to fix.
 (2) It's not clear if "make regress" should be run as a ordinary user
     or as root. Some tests need to run as root (regress/sys/fs/ffs)
     and "just fail" if run as non-root, while others need to run as
     unprivileged user (regress/lib/libc/clone) and silently (!!!!)
     skip important parts of the test (because it does not apply to root)
     if run as root.
 (3) library tests are only build and run dynamicaly linked on archs supporting
     that. Some should be compiled and run static + dynamic, because the
     PIC requirements show different paths in assembler code. There should
     be some generic framework to support this.

>How-To-Repeat:
Try to get a list of regression tests failing on your arch.

>Fix:
I see three solutions for the root/unprivileged problem:

 - add "just in time sudo" support (like pkgsrc)
 - split the "make regress" into three targtes: common, root, unpriviledged
 - make the tests not working for the current user prominently log the
   fact that they ran only a subset or are skiped completely

The latter requires some loging/summary framework though ;-)
>Release-Note:
>Audit-Trail:
>Unformatted: