It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years.

However there are other reasons (e.g. privacy) that might make it worth running locally for some people.

I have a Strix Halo and dual 32GB GPUs in my desktop, that sit idle right now, because the electricity to run them and to cool them in 110F weather Texas is currently experiencing pretty much nulls any savings I might see over getting better models from cloud providers. While I mostly use Claude or Codex with subscriptions for agentic work, for API use DeepSeek has usually been my go to, but now I guess it's GLM 5.3 or the Flash version. And, for security work that Anthropic or OpenAI models are likely to refuse, I've been using Kimi K3 (also via subscription, though their subscription is extremely stingy), but I guess GLM is now the one for that, too.

Anyway, yeah, even at the prices I spent on my local AI stuff (I bought before RAMpocalypse really kicked into gear, so I bought old server GPUs for about $350 each and the Strix Halo for a little over $2k) it was never going to pay for itself; I just like to tinker. But, I can't imagine spending today's prices for hardware for local AI.

When the memory shortage ends, I'll be down to the Apple Store (or, more likely, clicking refresh on the Apple outlet every few days). But, until then, there continues to be a glut of cheap and free models in the cloud that are better than anything I can run locally and they're faster, too.

So far I don’t regret buying an M1 Max device with 32Gb of RAM. The models available for it keep getting better (running just about okay for interactive use) and 400 GB/s of bandwidth is still considered a lot.

The models are currently improving much faster than the hardware and this doesn’t seem to have plateaued yet.

Jalapeno is matching or very near Vera Rubin at 1/4 the power. I would not buy hardware now.

That's basically the question I'm trying to answer.

If you're paying Anthropic or OpenAI to use their models, harness, governance, etc., I could see the local inference potentially coming out ahead. They're already starting to ratchet down what your money gets you on their platforms, and that can be expected to continue as the leaders of those companies continue to seek the road to the El Dorado that is being a trillionaire.*

If you're looking to get into the guts of AI development instead of having it handed to you by a provider, that's where it gets murky. I'm wanting to write some sort of agent that does things and get into making outputs consistent in the like, and I'm not sure whether to host something on GCP or buy an M5 Mac.

*Note: El Dorado is a mythical city and many people died trying to find it.

Tools vs services in my mind. There is no guarantee any provider will continue to do what they are doing for you at the price they are doing it.

The object permanence of not having to reinvent the world every time a model gets sunsetted has value.

Part of it is knowing that whatever sort of enshittification the cloud providers do, my local programming environment won’t ever be less effective than it is today locally. It’s the same reason my entire development stack from editor to compiler is open source. I don’t need to modify it today, but I always must retain the option to do so later.

There are several things I do in my life that only pay off in the event of a big disaster, like an extended internet outage, civil unrest, supply chain disruption, war, etc.

I like to be able to do the things I do even if offline for weeks.

I spent a lot of money for more flash in my iPad Pro so I can keep all of offline wikipedia and OSM in it, for example, along with tons of books. It’s sort of like being a digital prepper. (Being a prepper is a spectrum, from anyone who keeps food in their pantry to people building bunkers under their house - how much you invest is a personal prudence and threat modeling decision.)

Also, privacy. And when I got the Mac Studio the 512GB was only $15k, which is dirt cheap for that much VRAM.