alt Hacker NewsWhat do you need speed for? That's a genuine question, I feel like the limiting factor already is my creativity, attention span and budget. And I'm not even yet optimizing cost by batching things like review to slow local models over night, or schedule tasks to take full advantage of my subscriptions.
Replies
There are speed thresholds that open up new use cases.
Imagine speeding up current agents 10x, you switch from directing agents to pair-vibing on the fly.
Speed them up 10x more, and you get a SOTA model capable of analyzing and rethinking your entire file in between your key strokes. That would make for one hell of an autocomplete.
Pivot over application, going from coding to anything else, and this can easily give computers features previously impossible to make. In video games, fully general characters reacting realistically to arbitrary dynamic situations. In "serious" apps, interactive work with a system that understands your goals and adapts to you on the fly. Hell, even an OS that can tell you "hey, the data you're obviously looking for is in the tab over there, now highlighted".
And that's just tip of the iceberg. I'd personally love to explore the possibilities.
I just finished waiting almost four hours for Fable to write 700 lines of code, based on my three paragraph prompt. Some speed on these harder tasks would definitely be welcome!
It also spent almost 800k tokens on these lines…
You can get stuff done live on a call.
Humans have human attention spans and there's a lot of study going back decades for UX design because of this. < 100 Ms is instantaneous, 100-300 ms is noticable but still responsive. At 1 second, flow.gets interrupted, 2-5 seconds, you're clearly waiting, 5-10 attention wanders and 10+ seconds, you've lost them. The 0.1 / 1 / 10 second rule comes from Jakob Nielsen's HCI work. Perceived latency matters almost as much as actual latency, which is why chat interfaces drip out/stream words instead of just dumping out the answer at the end. At 750/tok/s, for Sol grade inference, it can spend 3 seconds on thinking tokens before outputting something to the user for a better answer while still feeling usable.
> attention span
IME waiting for an agent to work through a problem is a detriment to attention span; your mind drifts to other things while you wait. Maybe you can steer several agents in a round robin instead, but then there's a cognitive tax from context switching. Faster models mean fewer gaps in focus.