alt Hacker NewsBut with a local model with essentially free tokens, I can set it to work on tasks in the background without having to worry about any usage limits. Deferring simple bug fix tickets or other low complexity to an essentially unattended local model works just fine. And if the result occasionally fails review, it still didn't burn paid usage credits.
A local model tops 60 tokens/s max. Running it on the background means not being able to use on the task you are working on. When working on local hardware you cannot do concurrent requests. In deepseek for example you can have 100 agents running each getting at 120 tokens/s easily. In local models you are limited by your own infrastructure.