alt Hacker NewsFor the record, "the future" you are describing is running a tiny LLM from 2 years ago.
Those on-device weights are horribly outdated, underperform for their size, and cannot be updated to use a different model. I don't think Nvidia is afraid of Taalas.
Replies
That's a proof of concept created by a small startup. Obviously Intel or NVIDIA could do better. Given that it's ~50k tokens per second, even a lesser model opens up an entire universe of use-cases that we currently don't have access to, and that's locally.
There was a time when people were running less-capable minicomputers, but the future was mainframes.
There was a time when people were running microcomputers, but they were just toys; the future was mainframes and minis to do real work.
The way tech goes is that the "toys" become real. They become where the action is. The bigger stuff doesn't go away - there are still mainframes, even today - but the action moved on to microcomputers, and then to phones.
There will still be "big" models, but don't dismiss models on a chip because they are "toys".
Note well: I do not claim that models on a chip will certainly follow that trajectory. All I'm saying is that, given the trends in tech over the last several decades, don't dismiss them just because they are toys.
In the future, will a 2 year old model be useless? In 2 years? 10 years?