Re: CVS commit: src/crypto/external/bsd/netpgp/dist

[Julio Merino <jmmv%NetBSD.org@localhost>]

On 6/29/11 2:59 PM, Jukka Ruohonen wrote:
> On Tue, Jun 28, 2011 at 06:50:50AM +0000, David Holland wrote:
>> Perhaps if atf were less intrusive...?
> 
> What do you mean? I think it needs to be quite intrusive (sandboxing, etc.).
> Unquestionably the old /regress-style is not the way to go. Even if you
> dislike some parts of the API, already the consistency is a big win.

Yep, I had the same question here.  I don't know what the original
comment meant.

> On Wed, Jun 29, 2011 at 10:50:22AM +0100, Julio Merino wrote:
>> One of the ideas floating around in my head is to make atf-run (well,
>> kyua) support "foreign" tests.  The most basic form of this would be
>> programs that just return 0 on success or 1 on failure, but maybe it'd
>> be extended to also support autotest programs, etc.
> 
> Wasn't there already a test or two doing something like this? Fine by me,
> as long as we do not regress to the old /regress-style in which it is usually
> hard to even know what is being tested.

Some atf-ified tests do look like this, yes, and it's ugly.  But if we
could do this, we could lower the barrier of entry (specially for
third-party projects): you could plug in a simple test program that you
already have and get it to work inside atf, later refining it.

Also, using "foreign" test programs does not preclude the sandboxing.
The sandboxing is something that must be in place all the time IMHO
regardless of how the tests are written.

>> Sure, their functionality would be limited within the framework (as
>> compared to "native atf tests", but this would allow us to "mux" all the
>> tests in one single run without much effort.
> 
> But in reality, the gcc(1) test suite is probably the only one that would
> really be worth considering. NetBSD does not ship that much third-party
> software, and the availability of tests is generally quite limited among
> this group.

Maybe true, but I'd personally like to see all the available tests
running, even if they are not sophisticated enough.  (Otherwise, why are
we importing them at all?)

-- 
Julio Merino / @jmmv