alt Hacker NewsAmazon wasn't competing against open and free models that are starting to be good enough running on existing laptops.
OpenAI and Anthropic's moat is filling with cement faster than they can dig.
Amazon wasn't competing against open and free models that are starting to be good enough running on existing laptops.
OpenAI and Anthropic's moat is filling with cement faster than they can dig.
I have 128 GB of unified memory (M4 Max) and the user experience with local inference is still pretty bad. I'm so glad something like llama.cpp exists so I don't have to wrangle Python (which I hate), but OpenCode is entirely disrespectful of the KV-cache so I had to switch to Pi (but Pi is going relatively well actually).
Even so, I can't really run at hundreds of tokens per second which is practically table stakes for my work. Even if I did manage to run that fast, the model would probably be completely braindead and stomp all over the task.
Wish I could afford an M5 Max but I've been between jobs for months without even a single interview. Sucks to be a developer these days.