logoalt Hacker News

epistasis • today at 5:48 PM • 0 replies • view on HN

Inference has two steps, prefill and decode, with very different requirements on the compute to memory-bandwidth ratio.

Already these are run on different hardware, at least as I understand, and the computational needs could further drive differentiation.