Re: ATF tests still failing to complete within 2 hours on amd64

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Sat, Dec 03, 2011 at 04:04:51AM +0000, David Holland wrote:
> 
> Well, if it's broken we should certainly try to fix it. That said,
> even if it's not broken, I think three failures in 10,000 is too high
> for production use... at least if the failure causes a panic at boot
> time. Particularly a panic at boot time when new users are trying to
> boot the installer for the first time.

I can't see why.  One in 3,333 times, the test falses positive, the
machine reboots, and comes back up running normally.  No, I don't have
a power analysis to tell you what the chance of a type II error is -- but
I can tell you that I have personally seen this test reveal both severe
bugs in my RNG code during development, and hardware RNGs that fell off
the bus or were otherwise not producing random data.

If the machine is in fact working correctly and the code lacks severe bugs
that would impact system security, there is a 99.99% chance that the
machine will behave normally at any given startup, and a 0.01% chance that
it will panic and reboot, then behave normally (okay, okay, the conditional
probability that it will _not_ then reboot normally, but rather panic again,
is 0.0001%, assuming I counted my zeroes right...)  That seems entirely okay
to me -- and, given that the test is conditionalized on DIAGNOSTIC, those
who prefer an enhanced risk of false negative to the small risk of false
positive are free to do what they usually do: turn DIAGNOSTIC off.  What's
the problem?

Thor