tech-userlevel archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: long double losing mantissa bits
> Date: Tue, 19 Jan 2021 22:16:32 +0000
> From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
>
> This is not an integer, so conversion to uint64_t rounds it to
> nearest, with ties to even, so you get ufrac = 0x80000000000000 =
> 36028797018963968 as shown in the result.
Oops -- one more correction: conversion to uint64_t rounds toward
zero, not to nearest. The result happens to coincide in this case.
> It looks like you have an off-by-one error in your exponent handling.
> If you want to scale the fractional part into an integer, you need to
> multiply by 2^57, not by 2^56.
Another possibility is that you expected round-to-odd or something
instead of toward-zero or to-nearest -- or perhaps the code works as
intended but the test cases are buggy.
Home |
Main Index |
Thread Index |
Old Index