On Tue 19 Jan 2021 at 22:16:32 +0000, Taylor R Campbell wrote: > It looks like you have an off-by-one error in your exponent handling. > If you want to scale the fractional part into an integer, you need to > multiply by 2^57, not by 2^56. Thank you! Your analysis made me realise that I was miscounting the bits, and assumed one too many. So several of the test values were a bit too long and got rounded which confused me. So I did change the multiplication to 2^57. But the 2^56 was correct for the final format, so I just used it to get the lsb for rounding later, and then shifted it a bit right again. Final result is at https://gitlab.com/Rhialto/macro11/-/blob/master/parse.c#L386. -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG"
Attachment:
signature.asc
Description: PGP signature