A mess with complex arithmetics in earmv7hf

[Alexander Mukhin <alexander.i.mukhin%gmail.com@localhost>]

Dear colleagues,

I'm developing a Fourier transform library, and, while testing my code
on NetBSD/earmv7hf, found that even simplest complex arithmetics cause
the programs to dump core or return garbage, if they are linked against
libm:

	$ cat test.c 
	#include <complex.h>
	int main (void) {
		double complex z,w,u;
		z=1+I;
		w=2+I;
		u=z*w;
		return 0;
	}
	$ gcc test.c -lm
	$ ./a.out
	[1]   Bus error (core dumped) ./a.out

Tracking down the issue, I found that it is caused by the builtin
complex arithmetic routines, such as muldc3, taken from compiler-rt and
included in libm. These routines use the standard AAPCS calling
convention, while the compiler calls them via the VFP calling convention 
variant, utilizing VFP registers for parameter passing and result
return.

All this is because the complex arithmetic routines in compiler-rt are
defined with the COMPILER_RT_ABI macro, which expands to
__attribute__((pcs("aapcs"))) and overrides the default ABI.

The same applies to the complex arithmetic routines taken from
compiler-rt to libc.

A quick and dirty fix I used to check myself was to force the macro
COMPILER_RT_ABI to be undefined when compiling these routines for libm:

$ diff -u src/lib/libm/compiler_rt/Makefile.inc.orig src/lib/libm/compiler_rt/Makefile.inc
--- src/lib/libm/compiler_rt/Makefile.inc.orig  2016-02-27 19:06:56.000000000 +0000
+++ src/lib/libm/compiler_rt/Makefile.inc       2020-12-30 10:15:24.865722208 +0000
@@ -36,5 +36,6 @@
 .  else
 SRCS+= ${src}
 COPTS.${src}+= -Wno-error=missing-prototypes
+COPTS.${src}+= -U__ARM_EABI__
 .  endif
 .endfor

Surely someone more experienced would provide a better fix.

I opened a PR lib/55897 for the issue, but probably the port-arm list
is the better place to start with?

-- 
Kind regards,
Alexander.