>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?

That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.