From f5ce1664369edc7b1f213b6c7fbcf4e1259b7a7d Mon Sep 17 00:00:00 2001 From: Eitan Adler Date: Wed, 14 Feb 2018 07:59:30 +0000 Subject: [PATCH] msun: signed overflow in atan2 As a component of atan2(y, x), the case of x == 1.0 is farmed out to atan(y). The current implementation of this comparison is vulnerable to signed integer underflow (that is, undefined behavior), and it's performed in a somewhat more complicated way than it need be. Change it to not be quite so cute, rather directly comparing the high/low bits of x to the specific IEEE-754 bit pattern that encodes 1.0. Note that while there are three different e_atan* files in the relevant directory, only this one needs fixing. e_atan2f.c already compares against the full bit pattern encoding 1.0f, while e_atan2l.cuses bitwise-ands/ors/nots and so doesn't require a change. Closes #130 Submitted by: Jeff Walden (@jswalden github PR #130) Reviewed by: bde MFC After: 1 month --- lib/msun/src/e_atan2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/msun/src/e_atan2.c b/lib/msun/src/e_atan2.c index a4a985b5c03a..352db171d219 100644 --- a/lib/msun/src/e_atan2.c +++ b/lib/msun/src/e_atan2.c @@ -71,7 +71,7 @@ __ieee754_atan2(double y, double x) if(((ix|((lx|-lx)>>31))>0x7ff00000)|| ((iy|((ly|-ly)>>31))>0x7ff00000)) /* x or y is NaN */ return x+y; - if((hx-0x3ff00000|lx)==0) return atan(y); /* x=1.0 */ + if(hx==0x3ff00000&&lx==0) return atan(y); /* x=1.0 */ m = ((hy>>31)&1)|((hx>>30)&2); /* 2*sign(x)+sign(y) */ /* when y = 0 */