One of the hardest determinism bugs I had to solve on the PlayStation three was that the PPU and the SPU actually used a different instruction set and had a different internal floating point register size. We had a multi threaded physics simulation, and during instant replay, we had to ensure that the job scheduler sent the exact same work to the correct cores or we got back subtly different floating point values, which of course, immediately caused major divergences.

> I guess floats are still mostly deterministic if you use the exact same machine code on every PC.

Nope, they are not. Part of the problem is that "mostly deterministic" is a synonym for "non-deterministic".