alt Hacker News> Apple doesn't design GPUs on-par with Nvidia's efficiency yet
How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
> Apple doesn't design GPUs on-par with Nvidia's efficiency yet
How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM.
It's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.