logoalt Hacker News

bigyabai • today at 5:13 PM • 1 reply • view on HN

Apple's biggest bottleneck for real-world inference is prefill processing. They need a better GPGPU architecture, which is what I'm expecting M7 to reveal.

Replies

tedd4u • today at 5:28 PM

Agreed, seems like the M5 has already made steps in that direction, with 4x prompt processing / prefill performance vs. M4. [1] And token generation also got a 10% boost. The data below is only for Pro & Max but I think the base M5 got the same relative boosts vs. M4 base.

    Chip         BW (GB/s)   GPU Cores   Q4_0 Prompt   Q4_0 Gen
    M4 Pro (20c)    273         20          439.78        50.74
    M4 Max (40c)    546         40          885.68        83.06
    M5 Pro (20c)    307         20      ~1500 to 1700    ~56
    M5 Max (40c)    614         40      ~3000 to 3500    ~92
[1] https://www.hardware-corner.net/m5-pro-m5-max-local-llm-4x-f...