toolchain/58411: GCC/x86_64 10.5 and 12.4 miscompile GCC/sh3 12.4

[rokuyama.rk%gmail.com@localhost]

>Number:         58411
>Category:       toolchain
>Synopsis:       GCC/x86_64 10.5 and 12.4 miscompile GCC/sh3 12.4
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jul 10 06:50:00 +0000 2024
>Originator:     Rin Okuyama
>Release:        10.99.11 and netbsd-10
>Organization:
Internet Initiative Japan Inc.
>Environment:
NetBSD sakaizumii.local 10.99.11 NetBSD 10.99.11 (AMD64_NET_MPSAFE) #4: Wed Jul  3 14:55:32 JST 2024  rin@sakaizumii.local:/home/rin/src/sys/arch/amd64/compile/AMD64_NET_MPSAFE
>Description:
I've been troubled by wrong binaries generated by GCC/sh3 12.
This turned out to be, surprisingly enough, due to GCC/x86_64!!

If GCC/sh3 12.4 is built by GCC/x86_64 11.4 shipped with Ubuntu 22.04
(as well as our in-tree GCC/powerpc 12.4), it generates working GENERIC
kernel for landisk. On the other hand, if it is compiled by our in-tree
GCC/x86_64, it drops some `tst` insns by mistake:

https://mail-index.netbsd.org/tech-toolchain/2024/07/02/msg004458.html

and generated kernel does not work, of course.

For `cc --version --verbose`, `-mtune=nocona` appears for us and
`-mtune=generic` for Ubuntu, respectively.
(This comes from bsd.own.mk.)

If `env HOST_CFLAGS='-O -mtune=nocona' HOST_CXXFLAGS='-O -mtune=nocona'`
is used to invoke build.sh, GCC/sh3 built on Ubuntu miscompiles landisk
GENERIC in a similar manner to ours.

Also, if
`env HOST_CFLAGS='-O0 -mtune=generic' HOST_CXXFLAGS='-O -mtune=generic'`
is used, our GCC/x86_64 10.5 seems to generate working GCC/sh3 12.4.

However, unfortunately, `-mtune=generic` is not sufficient for
GCC/x86_64 12.4. Even if this option is specified, generated GCC/sh3
binary is broken.

Our GCC/x86_64 miscompiles insn-preds.cc (generated from sh.md).
If `#GCC pragma optimize ("O0")` is added to this file, generated
GCC/sh3 seems to work even with GCC/x86_64 12.4.

GCC/x86_64 uses `sete %al` (or equivalent) to prepare return values
from bool functions, *without* zero extension. As a result, return
values changes quasi-indeterministic depending on untouched upper
(24+32)-bits of %rax register.

What I've still not understood is why this bug does not affects us
more terribly. As far as I can tell, GCC/x86_64 mostly works fine;
it actually generates successfully working GCC for platforms other
than sh3.

Something wrong for GCC/sh3, or we've just overlooked other
serious problems :(
>How-To-Repeat:
Described above.
>Fix:
N/A