alt Hacker NewsThat video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
Replies
all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.