Re: Randomize configuration for automatic builds?

["Ian D. Leroux" <idleroux%fastmail.fm@localhost>]

On Mon, May 18, 2015, at 02:27, Kamil Rytarowski wrote:
> What do you think of introduction in the build machines randomized
> configuration of mk.conf(5) in each build?

While it'd certainly be nice if the non-standard options I use were more
regularly tested, I think the overall value of the automated builds and
tests is much higher if they are reproducible.  Randomizing the test
conditions might turn up some bugs that otherwise go unnoticed, but it
would make it impossible to narrow down which set of changes caused a
particular failure, because we could never be sure whether the
difference
between a successful build and a failed one was in the source code or in
the build options.

That said, I think we may be confusing two different things here,
the build clusters that provide up-to-date compiled binary
distributions, with the standard options, that people can just drop
onto a drive and try out; and the test machines that build and run
tests in search of regressions.  I don't think that randomized
configurations make sense for either of those, but the arguments are
a bit different in the two cases.

Now if we had enough resources to test non-standard configs
*systematically*, that might be interesting, but in my experience
breakage related to my non-standard config is so much rarer than
breakage/fixage in the standard builds that I doubt it'd be worth
the effort.

-- IDL