alt Hacker NewsI have a limited understanding of local LLMs, but from what I gather 128GB is not enough to match DeepSeek v4 Flash or GPT 6.1 Luna. And these models are so cheap right now. What's the use case besides having your data locally?
Replies
While personally I’m not using them for work, just tinkering, 128GB can run qwen 3.8 flash next at q4(~111GB gguf). That model seems to be quite capable from what I’ve read.
It’s quite evident that the api models can and have been nerfed at the provider’s discretion, then there’s the abliteration aspect too(api models refusing what they consider “harmful”) which makes the case for local.
Privacy, mainly. I don't see how it would make economic sense. $10k will buy an awful lot of GLM-5.3 tokens.
Being able to run your own uncensored model the way you want to.
"Having your data locally" is definitely the biggest one.
- The general consensus seems to be that the frontier models are kind of (understandably) "benchmaxxed" towards coding benchmarks and have suffered in some other areas like writing, art, etc. Lot of tweaked/specialized models on huggingface
- Getting around the censorship in commercial models
- While local models don't match frontier coding agents, you can run them for multiple 24/7 with a long running goal (The fans in Macs are presumably not built to run 24/7, but this can be mitigated to a considerable extent IME by simply pointing a desktop fan at it, and you can throttle things in software. On the discrete GPU side of things, you can usually get big power savings without much performance impact by throttling the GPU clock; things are usually bottlenecked on memory)
- Even small 27B/35B models are getting darn good, definitely more than powerful enough for classification etc... you don't need 128GB to run them
- Right now, providers like OpenAI are running at huge losses. At some point, they are going to need to turn a profit, at which point they may get significantly more expensive, and "hosting your own" may make more economic sense than it does today -- and be a valuable professional skill, even.
- Even more speculative: perhaps the future for many applications is training and running your own hyper-specialized models. Right now, training your own is nowhere near cost effective on self-hosted hardware vs. renting GPUs
Note: whether any of these use cases are "worth it" is up to the individual. I'm not trying to sell anybody on the idea. I don't personally do much local LLM stuff at the moment, but it's attractive and I've been close to pulling the trigger...