alt Hacker NewsSomeone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac.
It doesn't need to be the best. It just needs to be good enough. I'm not smart enough, but if I was I'd be working on a locally ran interface engine. That's where you charge say 100$ a year. Said engine would allow you to use a large open source model locally to the point where it's good enough to rival Haiku.
If I had to guess someone is working on this
Replies
Great comment, I'm working on my own Hermes/Mac Mini & the amount of sec ops/setting up its arms is a good challenge as a non-technical.
Building the iPhone-platform equivalent of this would be interesting.
The constraints will work against them long term. All this is doing is forcing people to look at how to run more with less. These companies are bloated beyond reality and have no incentive to optimize anything. They're starting out with access to everything and still producing very little tangible value comparative to the inputs. OAI and Anthropic represent the pure gluttony of the US tech bro market and it's not going to be the long term winner. It can't be lest the masses are cool at crowning these narcissists as kings.
> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?