That only works if you have all the unit tests and making sure the behaviors between different GPUs are expected, and can tolerate difference in implementation details such as numerical stability and complexity for some functions. AMD and Nvidia are doing vastly different computation for different implementation to the same algorithm for their APIs (can't tell due to NDA, but one algorithm Nvidia did it using broadcast, while AMD is using tile), and that's one of the reason I gave up. And perhaps only for light kernels doing basic BLAS operations and GEMMs. With a standard to follow, that's one of places where it indeed is giving it a good chance, for example on Tensorflow with ROCm or PyTorch. You basically have an insurance of the behavior and a translation layer for ROCm and CUDA by Python.
But anything other than that, nah.
Edit: Maybe I remember what algorithm that is. It is a trick to do coding theory stuff by using NTT (Number Theoretic Transform, the friend of DFT, the Discrete Fourier Transform, but on a ring and hence applicable to finite field) on GPU to do computation over a GF(2^8) field, for my former employment but I went back to using GFNI. Still I have a NDA to adhere to and can't really explain what happens.
That, or, you know. Ask Claude Code to write all of those unit tests.
Variance in numerical instability across hardware is basically a given, because all the different accelerators implement their "fast paths" in slightly different ways. Modern AI just takes it on the chin.
Anthropic, for example, already has to support both Nvidia CUDA and Google/Broadcom TPUs - AMD aside. And I expect those two to have more architectural and software differences than AMD ROCm and Nvidia CUDA accelerators would.
And yes. If you're wondering how they solved the problem of variance between those two: they didn't! Claude instances that run in the TPU land produce slightly different outputs than those in Nvidia racks! Just not different enough for, you know. Anyone to give a shit. No statistically significant effect on model performance.