alt Hacker News>Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Someone much smarter than me would figure this out.
One option would be an alternative bare metal OS which could allocate ram more carefully.
I'm sure with billions upon billions in VC funding a solution could be found.
Someone much smarter than me would figure this out.
Imagine if we had some kind of really intelligent AI model that we could pose problems like this to. It's obviously not the kind of thing an open weight model running on your local machine could do, but if we had some kind of really large, closed source model running in the cloud that could solve really difficult problems like this, I wonder if there'd be a market for it...