Re: etiquette for new failing test cases, followed by fix

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Thu, 19 Sep 2024 09:20:42 -0400
> From: Greg Troxel <gdt%lexort.com@localhost>
> 
> I have found a serious bug in reqmuo(3), and a fix in FreeBSD, validated
> by regression tests in proj and a hand-written test.  The problem exists
> in current and 10, and surely in 9 but I haven't checked.
> 
> We currently have no remquo test.  I've written one, stealing from
> tests/lib/libm/t_sin, and it shows some failing output.
> 
> I'm about to (assuming each step is ok)
> [...]
> Is this reasonable?  It will show an increased failing test for a bit,
> but the test really does fail, and my impression is that the xfail
> scheme is about keeping known-failing not-getting-fixed tests from
> obscuring "we just had a regression", which a few days of this won't do.

What you've done is fine, but what I do -- and what I suggest everyone
do, though we don't have any mandatory rule -- is:

1. File a PR to track this (and assign it to yourself if you're
   already working on it).

2. Commit an xfail test that demonstrates the issue and wait for the
   releng testbed to verify it (on relevant architectures).

3. Commit both (a) the fix and (b) removal of the xfail simultaneously
   in a single commit, and wait for the releng testbed to verify it.

4. Test pullups in a chroot on my laptop with `hg graft' or `git
   cherry-pick' and submit them.

I draft the commits offline in hg/git to test locally first, usually
on my laptop in a chroot -- following
https://www.NetBSD.org/gallery/presentations/riastradh/bsdcan2024/getstarted.pdf#page=46
-- or a VM, and then commit with `git cvsexportcommit'; this ensures
there's a good chance (2) and (3) will succeed on the first try.

For libm tests, step (4) is so quick for 9 or 10 that I submit pullups
for both anyway unless there's merge conflicts.

This way, after each step, even if I get interrupted by a conference
causing a thousand email messages to queue up while I'm busy, it's
easy to remind myself or anyone else how to pick up where I left off.

(And thanks for taking care of this!  Looks like we should add some
more automatic tests for edge cases and floating-point exceptions too
while we're here.)