alt Hacker NewsTo put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
Replies
This seems great. I have 128GB of unified RAM and a AMD Ryzen AI Max+ 395 that I can use but I feel I'm still not using it as well as I could...
The main/best setup I have running so far is qwen flash next with halogen-server.
But if anyone knows if and how one can run even more impressive stuff, please let me know!
That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
I've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.