Re: etiquette for new failing test cases, followed by fix

[Greg Troxel <gdt%lexort.com@localhost>]

Roland Illig <roland.illig%gmx.de@localhost> writes:

> The first commit demonstrates the buggy state by having a test that
> technically succeeds but has lots of FIXME comments in all the places
> that are wrong, already stating what is expected after the bugfix.
>
> The second commit fixes one of the bugs and updates the tests, removing
> several of the FIXME comments from the first commit. Further commits fix
> more of the problems, one by one.

Why do you propose a test that passes with a FIXME, rather than xfail?
Isn't that what xfail is for?

> The benefit of this approach is that each commit shows the changes to
> the main code together with their effects on the tests, and it provides
> a direct before-after comparison.

Doesn't xfail do that too?

> The downside is that while there are still FIXME comments in the code,
> the tests already succeed. In that phase, the tests only demonstrate
> that the bug is still there, just in case a refactoring commit slips in
> between the bugfix commits and accidentally changes the behavior. In
> this stage, the tests don't ensure that the code behaves as reasonably
> expected, but this stage is usually short.

The possibility of some refactorting accidentally fixing the bugs in
question are approximately zero, unless someone does a "merge all fixes
to libm from FreeBSD".  That hasn't happened in 12 years, and someone
seeing this discussion, deciding to do it, and not talking to me about
it seems vanishingly unlikely.

>> I'm about to (assuming each step is ok)
>>
>>   - do a build and then anita run of current with my new test, and
>>     without the fix
>
> This sounds as if the current buggy state will not be available via the
> version control history, so to see the changes in behavior, anyone
> interested would have to check out, build and run the buggy code
> themselves to see what the _change_ was.

I don't follow that.   When I wrote, remquo was buggy (as is has been
~forever) and there was no test.

Now there is a test, and it fails:
  https://releng.netbsd.org/b5reports/amd64/2024/2024.09.20.07.50.38/test.html#lib_libm_t_remquo_remquo_args

After I commit a fix, that will get shorter -- it's the middle failure
that causes real trouble -- and then the second fix (not from FreeBSD)
should resolve the other two.

I don't follow "to see what the _change_ was".  There will be a bugfix
commit, with a comment.  How is this hard, and how is it different from
what we usually do?