If Nvidia didnt abandon consumer GPUs we all could be running better models locally by now. The moat is companies like Nvidia not releasing better consumer hardware so they can partner with bigger companies so they can sell us overpriced hardware.
They release a new RTX series every two years, last time was 2022. (edit: I mistakenly quoted the year wrong, not sure why, but it seems obvious to most that Nvidia doesn't care about the local consumer market anymore)
Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.
Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.
GPU-style platforms (lots of compute connected to a big shared pipe that connects to a large pool of fast memory) are inherently more efficient for centralized inference.
Local inference gets truly interesting when the proper next-gen, natively AI and not just a repurposed GPU, stuff starts to land. Initially things like compute-in-memory and fancy flash, and later possibly the truly exotic stuff like various RMM (read-mostly-ram, for example stacked FeFET on logic) tech.
IMO there is no chance for ubiquitous local AI to happen if that means people using GPUs. But the industry is chasing all possible solutions for more efficient inference platforms, and if something like pachinko machine architecture with FeFETs ends up winning, suddenly buying a local ai inference solution in a box is no longer as insane.
If consumers were willing to pay more for GPUs like datacenters do, NVIDIA would have sold plenty of consumer GPUs. Or would you press a button to halve your revenue for some reason?
(I say this as someone who would have liked a 5090...)
> Hoping AMD steps up to the plate
Nvidia has nothing close to the Strix Point, Strix Halo, Gorgon Halo, etc.
My GPD pocket 4 handheld can run Qwen 3.8 27b only because of AMD making embedded mobile inference cheap.
> Hoping AMD steps up to the plate
AMD's aggressive marketing as an AI-first company leaves me pessimistic about that.
I'm running qwen3.8-flash-next at 4-bit quantization on two AMD Radeon AI PRO R9700 cards at 75 tok/s. And I can actually run two models in parallel at around 50 tok/s, so you can get by with only one card. I switched from NVidia to AMD for a fully open-source stack for my local inference.
AMD Radeon AI PRO R9700 is $1600 right now (I bought them last year for $1300).
They are also quite power-efficient. My setup typically uses around ~300W during inference.
No gut feeling necessary. They announced weeks ago that they are going to be $1200-$1800 (Xbox specifically, PS6 somewhere in that range too).
Yes, ten the home AI market would be similar to the serious PC gaming market used to be when they were the main market for the latest GPUs.
> we all could be running better models locally by now
Apple silicon says "hold my beer", which I know because i've been doing this on my 3 year old quite beefy mac studio, for example. it's good enough for many tasks.
Not defending Nvidia here, but the RTX 50 series was released in 2025, was it not?
Regardless, they aren't planning on sticking to the two year release cadence any more which does suck.