logoalt Hacker News

wez470 • today at 5:38 PM • 2 replies • view on HN

How cost effective had it been? I don't want to rely on 3rd parties either, but my experience is it's super expensive to run locally versus using a provider (both in terms of hardware costs and ongoing electricity usage).


Replies

lrvick • today at 5:56 PM

No third party can ever provide the privacy and sovereignty of being able to run locally with abliterated models. I would do it this way even if the providers were much cheaper as I value those properties a -lot-.

In practice though, I can produce about 3B tokens/month of Qwen 3.8 Flash Next which is the model I crank almost 24/7. That would cost something like 1k/mo on OpenRouter. My setup costs about $300/mo in electricity, though getting solar soon to drop that to zero.

➕ show 1 reply
ahartmetz • today at 7:31 PM

Running locally is not a problem if you allow it to be slow. I get about 3.5 tps with Qwen3.8-27B:UD-Q4_K_XL on my 64 GB laptop and about the same tps with Qwen3.8-Flash-Next:UD-IQ4_XS (IIRC, or UD-Q4_K_XL) on my 128 GB desktop with a 16 GB GPU.