logoalt Hacker News

lmtlssss • today at 7:38 PM • 0 replies • view on HN

We measured how far off the float_pkg results are, by comparing the returned float32 and float64 values with exact arithmetic and expressing the distance in units of the last place. The cut is 0.5 of a unit for correct rounding, and sqrt(2.0) in double falls at 0.565, with the correctly rounded neighbour at 0.435. So even that one-unit error is beyond the rounding boundary.

But the misses extend much further than that rounding boundary. Across the same 10,000 calls per operation we measured 19,147 wrong results; 7,213 of those are within one unit, while 3,775 are more than 1,000 away. 4,063 have the wrong sign and 3,370 confuse a number with infinity or NaN. The reciprocal result is useful here: none of its 384 wrong float32 results is a near miss, with 294 returning infinity or NaN for a number, 66 the wrong infinity or NaN, and 24 getting the sign of zero wrong.

The table is in "How far off"; python3 exact.py sizes vectors.txt results.txt prints it.