logoalt Hacker News

Pete warden on local AI, open models, voice interfaces, the future of hardware

178 points • by datascientist • today at 3:38 PM • 159 comments • view on HN

Comments

lrvick • today at 5:31 PM

I run 8x r9700s at home which lets me run up to 2x 1M context sessions of GLM 5.3 flash at ~50tps or 12x sessions of qwen 3.8 flash next at ~100tps.

I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.

There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.

I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

➕ show 7 replies
giancarlostoro • today at 3:59 PM

If Nvidia didnt abandon consumer GPUs we all could be running better models locally by now. The moat is companies like Nvidia not releasing better consumer hardware so they can partner with bigger companies so they can sell us overpriced hardware.

They release a new RTX series every two years, last time was 2022. (edit: I mistakenly quoted the year wrong, not sure why, but it seems obvious to most that Nvidia doesn't care about the local consumer market anymore)

Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.

Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.

➕ show 7 replies
paxy • today at 4:33 PM

I don't see any difference between the conversations around self-hosting local AI and self-hosting regular software, which have been going on for decades.

Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.

People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?

➕ show 16 replies
herf • today at 4:37 PM

I love local AI, but even so, I admit the economics point the other way. Unlike gaming, AI is a uniquely good match for data centers because the applications today aren't latency-sensitive at all (except perhaps DLSS/self-driving/robotics). Most importantly, utilization is massively better with always-on servers doing parallelism (in-flight batching). Also in some areas (like California) our local electricity is quite expensive, so moving the work 100ms away can help. The privacy implications are horrifying, though.

➕ show 3 replies
brendan_rules_t • today at 4:59 PM

I have a 128gb ddr5 unified system, that i have been setting up with linux and local llms and other models and i have to say it feels beyond next level! I still use claude, grok, codex, and neo clouds to offload heavy work loads, but anything i want private or says too much about my system is now done locally. The voice agent experience is incredible

montroser • today at 4:56 PM

I run Qwen3.8-27B Q4 on a $1500 box in the basement with 32GB VRAM, and a customized OpenWebUI on top. It's pretty great. Occasionally I consult Claude for a second opinion, but I'd say I've moved 95% of my previous conversational Claude usage to be local instead. I use it for leisure coding projects too, but for real work, it's still DeepSeek 4.1 Flash for now.

cj00 • today at 5:24 PM

Invariably when local AI comes up, people try to see how it can be used for frontier-level AI tasks like asking it to one shot a web app from scratch and then they get disappointed when they discover it's not performing at the same level as Opus.

But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.

And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.

➕ show 1 reply
smath • today at 5:34 PM

What is a good place to track the latest capabilities of open source local LLMs vs commercial models? Things like overall intelligence performance, latency/speed, hardware needed, cost.

Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.

1970-01-01 • today at 6:40 PM

Voice interface is the "next big hit" as it is removing the very last hurdle to accessibility. Anyone that can talk and hear will be able to use AI. No screen, no keyboard, no problem. This was supposed to be the home assistant in a nutshell but we all know how limited it was in reality.

➕ show 1 reply
sachaa • today at 5:43 PM

Why should users have to choose a model at all? We don't choose a codec before playing a video. An app should be able to request transcription or summarization, and the browser should find and run a suitable local model. That's the layer i am exploring with modelxare.com.

jkingsman • today at 4:19 PM

Even dumber local models that are heavily quantized to fit on consumer GPUs can be lovely as a dimwitted household assistant. A lot of people look to offload things they already do to LLMs, which can be great with frontier models, but you've got the highest standards and preferences for tasks you already do, so it's the most likely to fall into a "should have just done that myself" outcome. I've found great success in instead lowering my threshold of things I'd take time to do -- I wouldn't spend time on it, but I'd spend 60s prompting an LLM to take a crack. I'm fortunate to have the hardware to run Qwen 3.8 Flash Next at q8, but most of the tasks I have it do would be totally doable for a mid-quant Qwen 3.8 27b.

Some recent tasks I've thrown it:

* Go through my blood test data (from EHR XML dumps from prior years) and give me some over-time charts for consistently-sampled measurements

* Go through my Good Articles bookmarks, epubify them, and drop one down onto my ereader each morning for my commute, then archive the offline version into cold storage

* Go through my music library, identify my most-listened low-bitrate tracks, and source or assemble a shopping cart to get high res versions

* Analyze local tax/foreclosure auctions for vacant land; grab aerial screenshots, tax data, ownership data, and compile a weekly report of things maybe worth bidding on based on some specific selection criteria (fun facts: if you're willing to accept some headaches, 5+ acres can be yours for under $5k, if you're in the right area)

* Dump my grocery store loyalty data, analyze my purchasing habits, and prepare a weekly instacart for my approval that aims to sync with usage habits. Tweak and learn volume/order adjustments over time.

* Given a KVM that can mount ISOs onto the host, reimage two old laptops of unknown status with Debian and install Minecraft so my nieces and I can have a LAN party

* Organize my music (high confidence MusicBrainz Picard matches), movies/TV shows, ebooks, and video game ROM collections with metadata and good folder organization

* Archive bookmarked fanfiction for my offline library (slowly, and respectfully, and with an actual browser. The browser is anti-bot-countermeasured to the gills, but QFN regularly solves hCaptcha and Cloudflare turnstile challenges. Once I gave it the ability to drag, it even reasonably reliably solves drag-shape captchas. A little while back it made a purchase on eBay which is NOT a bot friendly site.)

Would I personally spend time on doing most of these? Generally not. Could an LLM screw them up so badly I regret it? Also generally not. I get the luxury of a psuedo-assistant that doesn't judge how menial the tasks I send it are, and I save time. With a local LLM, my privacy concerns are far less than with frontier models, and local abliterated models are less (or just not at all) twitchy around things that frontier classifiers balk at.

I'm hopeful that big projects like Muse can help to normalize menial task offloading, and that local LLMs provide the bridge for more technically minded folks to use them even more aggressively and limitlessly (as in compute limits/subscription costs) and cheaply.

➕ show 1 reply
maxdo • today at 4:05 PM

i'll share a lonely voice on hn, i care very little about local ai.

I feel like people who consider it as an option thinks of narrow scenarios vs ai is part of our life like internet is. How do you things on mobile for example with your local models?

AI is very competitive, I use various models, I like to be on cutting edge and to be productive, its the opposite from doing something permanent, and not changing, where local models can be good. I like to do things on the go, and the last thing any normal person would want is to have DC at their basement.

➕ show 15 replies
philipodonnell • today at 4:18 PM

I think one of the more compelling reasons for personal-local AI is that when two services you use add their own AI that only works in that service it creates the need for a third AI to bridge those two. But any service the proposes to sell you that third AI will be blocked by the other two, thus, only people who can create their own local service will be able to bridge them, and that will be the new version of DIY.

xnx • today at 4:55 PM

"Local" is the wrong distinction. Self-hosting doesn't have to mean running off a box in my house. I don't care about using my own hardware as long as I can run the AI anywhere I please.

➕ show 1 reply
martialwave8 • today at 7:38 PM

[flagged]

Farooqiainews • today at 7:33 PM

[flagged]

rockingmeister • today at 5:13 PM

[flagged]

traydstar • today at 6:10 PM

[flagged]

asifniaz_ai • today at 6:52 PM

[flagged]

erinnakamato • today at 6:36 PM

[dead]

dahunt33 • today at 4:35 PM

[flagged]

gnaneshwar • today at 4:59 PM

[dead]

Betelbuddy • today at 4:53 PM

[dead]

routebraid • today at 4:40 PM

[flagged]