alt Hacker NewsIt's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
Replies
redox99 • today at 9:25 PM
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
➕ show 3 replies
IgorPartola • today at 9:24 PM
My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.
➕ show 2 replies
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.