alt Hacker NewsThis is a temporary problem. A 10x-100x speedup will allow us to simply wait for the response, instead of finding something else to do in the meantime.
I wonder what the experience is of people here with the ultrafast mode (8x) of Codex.
Replies
Speed is the real solution here.
I've tried to find a combination of a good-enough model, a VERY fast provider, and cheap enough serving to make this all doable. DeepSeek itself is quite fast, but I crave even more speed (and ideally something hosted outside of China).
Fireworks and Baseten seem to be the leading providers that are available to a solo dev like myself, and support any reasonably decent modern models (DeekSeek 4.x, GLM 5.x, etc). Fireworks has been a real dud, but Baseten in FAST mode has been absolutely amazing. Not cheap - but I find it substantially cheaper than Terra / Sol (low), etc. I haven't found Haiku to be smart enough to be useful.
DeepSeek (the provider) and beyond seem to hit the sweet spot to keep me mostly in-context, so I don't need 4 parallel work items to feel productive. So roughly 200tps or greater. Vanilla Baseten is ~400 tps, and FAST is 700-800 avg.
Price wise, Deepseek hosting is extremely cheap, Baseten is a step up - maybe 5x or more, and FAST mode is well beyond that. But I still find myself dual wielding Deepseek and Baseten FAST when I need the speed.
The other benefit: for my own work in very limited hours, I'm more than happy to pay for answers to come back faster.
Unless we get some kind of very different architecture I’m not sure we’re going to get a 10-100x improvement any time soon, api models will always preference throughput over individual task latency, so any improvement that sacrifices serial decode for better parallelism is going to be more economically viable.
While the LLM speed will increase, some stuff will be incompressible: running unit tests, tool use, UI interaction / mouse control. In my case, a significant amount of the is already spent on that, so much that I rarely ever select fast mode.