For me, a HUGE benefit to running local models is that my data stays mine, on my computers only.

Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business.

With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.

For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams).

PS: I wouldn't be surprised if Microsoft runs something similar on their end :)

But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.

The M3 Ultra was a lot cheaper for most of its lifetime, and really the main reason for buying it is if you want a lot of unified RAM, to run bigger models than 27B models.

The new M5 Ultra should deliver ~50% faster token generation (1.2TB/s mem bandwidth); and extrapolating from my M5 Max (since the M5 Ultra is literally just 2x Maxes), probably ~3x faster PP.

But I don't think it's fair to look at this only from monetary ROI vs API. With local models, you get privacy and ownership.

I do not trust _any_ API provider with my most personal information; such as for example, all my messages, emails, daily journals spanning a decade+, all my photos and videos, etc. So it unlocks new use cases that I simply don't feel comfortable with via API.

And a personal assistant with ALL my context and data, locally, has been incredibly useful for me :) Zero outages either, zero "overloaded", etc. Nearly-zero refusals too (I don't run abliterated models; thinking prefill has worked for anything I've wanted to do)

I have 2x 3090s and I get 260 tokens/sec peak (110 avg) with Dflash2 and about 1500t/s prefill with a Q4 quant of Qwen 27b. I didn't buy an overpriced Apple product and it performs much better. It's extremely reliable for Agentic coding and I can run 2-3 simultaneous agents with a full ~260k context window.

I realize the price of NVIDIA has gone up but there are plenty of GPU options from others like AMD and Intel with reasonable performance.

A slow AI tasks that can process my self hosted personal journal, my medical history in fasten, my diet and exercise in Mealie and Sparky Fitness to offer insights once a day or even once a week is better than nothing. Because I would never upload that data to any of the AI companies.