alt Hacker News> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Replies
One way would be to generate a new model for every request. Currently we create an entire universe for each step when training if we predetermined what needed to be created it could be much smaller.
> like “make a cure for <disease>” that will yield billions of dollars of value
I think it's important to qualify that while I will very much appreciate such an innovation and I think it'll provide significant value to society... I remain doubtful of the ability for any company to actually capture such an astronomical yield. Either there's enough of an investment required by the company to help get that to market that the yield will be significantly lower or everyone and their brother will just launch competing products two days later. And, as a society, I don't think it's logical for someone writing "Hey Alex, cure cancer" with billions of dollars so I doubt there's going to be much appetite for creating such a moat via legislation.
This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox. It very well may be that AI tools can accomplish the labor that used to be required to create such a business, and I'm sure a few people will land a jackpot and get rich in the transitional phase but if it's so cheap to build it why would someone pay you money for yours instead of just running the same prompting themselves? For most people that's because they lack the knowledge that they can but you'll still have a significant number of instant impersonators.
>Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Someone much smarter than me would figure this out.
One option would be an alternative bare metal OS which could allocate ram more carefully.
I'm sure with billions upon billions in VC funding a solution could be found.