The biggest issue was actually that the Mac Pro was designed specifically for dual GPUs- in the era of SLI this made some sense, but once that technology was abandoned it was a technological dead-end.

If you take one apart you'll see why, it's not the case that you could have ever swapped around the components to make it dual-CPU instead; it really was "dual GPU or bust".

Somewhat ironically, in todays ML ecosystem, that architecture would probably do great. Though I doubt it could possibly do better than what the M-series is doing by itself using unified memory.