Re: etiquette for new failing test cases, followed by fix

[Roland Illig <roland.illig%gmx.de@localhost>]

Am 19.09.2024 um 15:20 schrieb Greg Troxel:
> We currently have no remquo test.  I've written one, stealing from
> tests/lib/libm/t_sin, and it shows some failing output.

When I'm in such a situation, I usually do a two-commit approach:

The first commit demonstrates the buggy state by having a test that
technically succeeds but has lots of FIXME comments in all the places
that are wrong, already stating what is expected after the bugfix.

The second commit fixes one of the bugs and updates the tests, removing
several of the FIXME comments from the first commit. Further commits fix
more of the problems, one by one.

The benefit of this approach is that each commit shows the changes to
the main code together with their effects on the tests, and it provides
a direct before-after comparison.

The downside is that while there are still FIXME comments in the code,
the tests already succeed. In that phase, the tests only demonstrate
that the bug is still there, just in case a refactoring commit slips in
between the bugfix commits and accidentally changes the behavior. In
this stage, the tests don't ensure that the code behaves as reasonably
expected, but this stage is usually short.

> I'm about to (assuming each step is ok)
>
>   - do a build and then anita run of current with my new test, and
>     without the fix

This sounds as if the current buggy state will not be available via the
version control history, so to see the changes in behavior, anyone
interested would have to check out, build and run the buggy code
themselves to see what the _change_ was.

All this is assuming that there's no undefined behavior involved, as in
that case, the buggy code may not behave deterministically, making it
hard to make the tests that demonstrate the bug succeed reliably.

It's just an idea, I'm also fine with the approach you suggested.

Roland