Inference has two steps, prefill and decode, with very different requirements on the compute to memory-bandwidth ratio.
Already these are run on different hardware, at least as I understand, and the computational needs could further drive differentiation.