alt Hacker NewsThat's a proof of concept created by a small startup. Obviously Intel or NVIDIA could do better. Given that it's ~50k tokens per second, even a lesser model opens up an entire universe of use-cases that we currently don't have access to, and that's locally.
I don't think it's obvious. The 8B model that the Taalas cards run aren't any larger than what a modern smartphone can inference with GPU compute shaders. Users don't need SRAM or dedicated hardware to get interactive tok/s at that scale.
To push beyond the envelope of what a mobile GPU can do, you'd have to build gigantic coprocessors for 30-100B models, which would probably cost more than the phone/laptop itself to integrate on-die. Nevermind the fact that SRAM is several times more expensive than DRAM gig-for-gig.