logoalt Hacker News

IshKebab • today at 6:40 PM • 2 replies • view on HN

... by one ULP. Anyway I don't think anyone is just feeding a raw sqrt into their synthesiser.


Replies

loeg • today at 6:53 PM

> For a normal argument it ends on the root or one unit beside it: in the vectors above it is one unit off for 607 of 9,156 normal float32 arguments and 1,146 of 9,236 normal float64 ones, and never further. For a denormal argument the start is too far away for that many steps: 128 of 359 float32 denormals are wrong, 104 of them by more than two units, and sqrt (2.0*-149) returns 11 times the root.

lmtlssss • today at 7:39 PM

[dead]