[src/trunk]: src/lib/libm/src Fix signed overflow in atan2

[eadler <eadler%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/f48698b24dec
branches:  trunk
changeset: 831075:f48698b24dec
user:      eadler <eadler%NetBSD.org@localhost>
date:      Sat Mar 10 09:44:47 2018 +0000

description:
Fix signed overflow in atan2

As a component of atan2(y, x), the case of x == 1.0 is farmed out to
atan(y). The current implementation of this comparison is vulnerable
to signed integer underflow (that is, undefined behavior), and it's
performed in a somewhat more complicated way than it need be. Change
it to not be quite so cute, rather directly comparing the high/low
bits of x to the specific IEEE-754 bit pattern that encodes 1.0.

ok martin@ pgoyette@ maya@
obtained from FreeBSD

diffstat:

 lib/libm/src/e_atan2.c |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (21 lines):

diff -r 57fa27095aae -r f48698b24dec lib/libm/src/e_atan2.c
--- a/lib/libm/src/e_atan2.c    Sat Mar 10 09:37:59 2018 +0000
+++ b/lib/libm/src/e_atan2.c    Sat Mar 10 09:44:47 2018 +0000
@@ -12,7 +12,7 @@
 
 #include <sys/cdefs.h>
 #if defined(LIBM_SCCS) && !defined(lint)
-__RCSID("$NetBSD: e_atan2.c,v 1.12 2002/05/26 22:01:48 wiz Exp $");
+__RCSID("$NetBSD: e_atan2.c,v 1.13 2018/03/10 09:44:47 eadler Exp $");
 #endif
 
 /* __ieee754_atan2(y,x)
@@ -67,7 +67,7 @@
        if(((ix|((lx|-lx)>>31))>0x7ff00000)||
           ((iy|((ly|-ly)>>31))>0x7ff00000))    /* x or y is NaN */
           return x+y;
-       if(((hx-0x3ff00000)|lx)==0) return atan(y);   /* x=1.0 */
+       if(hx==0x3ff00000&&lx==0) return atan(y);   /* x=1.0 */
        m = ((hy>>31)&1)|((hx>>30)&2);  /* 2*sign(x)+sign(y) */
 
     /* when y = 0 */