alt Hacker NewsLocal inference can't arrive soon enough. On my inferior hardware the models still struggle to build the correct JSON to be able to call tools. Once either I get better hardware or the local models improve I'll happily leave it chugging away for a few hours to arrive at a solution because I know I'm not going to get token shock from all the looping if I'm running local inference.
I have spent way too much on hardware. I’m sure I will look back on that after the eventual price normalization with some regret. I do think we will look back at this shortage and name it as the thing that accelerated innovation at getting quality out of smaller models.
Hang tough. The other day someone posted a solution to get a model running off of SSDs. It won’t be fast, but it will be coming.