alt Hacker NewsGLM 5.3 is probably the sweet spot open weights model if you want to go beyond deepseek flash or the new glm flash. I used it with pi and had a fairly good time, especially since it’s less touchy about cyber and whatnot than the US guys. It’s slightly behind Kimi in ability but it’s a lot easier to run it, I’d expect prices (and speed!) from third parties to be noticeably better.
Assuming you’re willing to drop a fat stack of cash on the upcoming Mac m5 ultra with 512 gb unified memory, you can even run it locally, quantized to 4 bit. Whether it’s even slightly reasonable, well, my wife would probably skin me alive but maybe yours is more understanding.
Replies
In terms of pure tokens per dollar, absolutely not worth it.
That said, when I bought my pair of Sparks, the best model I could run on it was GPT OSS 120B. That has an AA score of 24.
Today, the best model I can run on them is GLM 5.3 Flash at Q4, AA score 57. Just still out on GLM 5.3 mixed quant.
So from that perspective, they are many times better value than when I bought them, and will likely continue to increase in value.
One could also run it locally on a used dual xeon (or amd-equivalent) server with 512GB RAM, albeit slower, if you have a useful workflow for it that's like "take this day's efforts and run it through various analysis agents", combined with giving it one-shot tasks/modules to build overnight. You would want a place like a garage or basement to put the server because it'll be loud.
I have just built an Epyc with 512gb DDR4 3200 RAM for a "reasonable" price and I'm hoping to have a setup with GLM as the architect and Qwen 27b/Next Flash as the implementer. This is 1/5 of the price of the Mac, but also probably 1/5 of the speed lol.
Its reasoning leaves a lot to be desired :(
Though I appreciate how good it is at "solid" grunt work and at that price (in fact I am paying the grandfathered subscription price; mostly).
I am planning to let go for my Claude AI subscription which I now use only for "planning" and maybe use that via Open Router as PAYG (at to try how it ends up). But god glm is bad at "talking" and "responding" anything prose. Not only quality but it's almost impossible to tune it and make it let go of its habits and biases and enthusiasms which often result in too many too and fro.
So I sometimes wonder at what point that starts becoming the cost and mental hassle. Maybe it's not there for me yet.
Well if you did get the m5 ultra could you obliterate the guardrails and then your wife can ask it pertinent but unsafe questions about how to punish you. Seems doable.
When we consider:
* LLM usage is new for the world
* Models are evolving quickly with high worldwide competition
* Hardware is evolving despite RAM shortages
Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what you’d spend otherwise on serverless or dedicated GPU spend…