We're maybe only 2 years away from really useful, relatively affordable local LLM usage. You can buy a 5090 PC for $5-6k but 32GB of VRAM really limits model sizes to ~31B. And that won't change (even with NVidia's next generation) because NVidia uses VARM as an aggressive market segmentation technique.

No, the hope really is these other platforms with a shared memory architecture. The DGX Spark won't be it because of the aforementioned market segmentation. So that leaves two players: AMD and Apple.

The AMD platform is still too low memory bandwidth, currently <300GB/s. For comparison a 5090 (or 6000 Pro) is 1.8TB/s and the M3 Ultra Mac Studio is ~900GB/s. Oh and B100/B200 uses HBM3e memory at ~3.2TB/s. The M5 Max in some Macbook Pros tops out at ~600GB/s. So you need access to better RAM and better CPU architecture for all this.

My great white hope is Apple. They have the market power to get memory and build silicon that coul dhave enough FLOPS to compete with NVidia's platform. They've started talking about it and I've seen rumors they're targeting the M7 generation (2028) for a huge leap. I'll believe it when I see it however.

But the point is, I think we'll be running 31B models at 100+tok/s on enthusiast hardware in 2 years and we'll likely be able to locally run 100-400B models, possibly larger.