alt Hacker NewsLocal AI sounds nice but most of Apple’s PCs and other devices don’t come with enough RAM for a decent price needed for good model performance and macOS itself is incredibly bloated.
Replies
SchemaLoad • yesterday at 11:37 PM
Depends what you are actually doing. It's not enough to run a chatbot that can answer complex questions. But it's more than enough to index your data for easy searching, to prioritise notifications and hide spam ones, to create home automations from natural language, etc.
Apple has the ability and hardware to deeply integrate this stuff behind the scenes without buying in to the hype of a shiny glowing button that promises to do literally everything.
➕ show 1 reply
That's true for current LLMs, but Apple is playing the long game. First, they are masters of quantization optimization (their 3-4 bit models perform surprisingly well). Second, Unified Memory is a cheat code. Even 8GB on M1/M2 allows for things impossible on a discrete GPU with 8GB VRAM due to data transfer overhead. And for serious tasks, there's the Mac Studio with 192GB RAM, which is actually the cheapest way to run Llama-400B locally