logoalt Hacker News

lrvick • today at 5:31 PM • 9 replies • view on HN

I run 8x r9700s at home which lets me run up to 2x 1M context sessions of GLM 5.3 flash at ~50tps or 12x sessions of qwen 3.8 flash next at ~100tps.

I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.

There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.

I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.


Replies

lrvick • today at 7:34 PM

For those asking in threads. 4x r9700s will cost about 6-8k depending on how patient you are with ebay auctions.

From there any motherboard and CPU that can give you 4x slots of 16x Gen 5 PCIe slots positioned such that you can install 4x cards is probably fine. Ideally with a CPU capable of memory encryption.

My latest pairing is a ASUS Pro WS W890E-SAGE SE For $800 open box, and a Xeon 654 for $1300. My first build was an AMD EPYC rig for similar cost a few months ago using a Gigabyte ME03-PE0 and a AMD EPYC Siena 8224P for about 2k combined. I specifically wanted one amd and one intel machine for reproducible build diversity which most probably do not care about.

Do not get too attached to specific cpus and boards. Get what is available for cheap with the needed pcie bandwidth open box on ebay from reputable sellers to save thousands of dollars.

Be sure to get a power supply that can handle all 4 cards. I run Silverstone Hela 2500s at about $800. Do not go cheap on power supplies. It will end in pain. Can confirm.

Then 2-3k for 64gb in DDR5 RDIMMs. This is the most painful thing to purchase but it is what it is.

Not only do I recommend this setup, I plan to build more of them to share with my team.

If you are patient and buy used, a 10k budget can get you a very powerful local AI rig running 100% FOSS software/drivers even today.

back_to_basics • today at 5:56 PM

>I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

Which is exactly why said power and utility will soon be BANNED under a myriad of false aegis'.

An economy that almost exclusively extracts ECONOMIC RENT from the working class, won't permit any "leakage" or autonomy.

➕ show 1 reply
MomsAVoxell • today at 6:24 PM

I exclusively only use local models. I do not subscribe to anything, and I have my agents wrapped and inspected.

2xDGX Spark cluster is my current load, it is sufficient for a great deal many wonderful, wonderful things. It isn't fast as consumer AI, but I have nevertheless been able to do wonderful things with it for the past year'ish.. I'd get a couple more, but indeed they've tripled in price, so yeah .. things will just do for now, and it's fine. Pretty great, actually.

I think the way we control AI is we air gap it. My cluster is air-gapped.

txdv • today at 5:40 PM

whats electricity bill and do you not fear that your hardware will be obsolete while others will have access to better models on faster hardware?

i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.

I have to say, no limits stopping sounds nice though.

➕ show 1 reply
wez470 • today at 5:38 PM

How cost effective had it been? I don't want to rely on 3rd parties either, but my experience is it's super expensive to run locally versus using a provider (both in terms of hardware costs and ongoing electricity usage).

➕ show 1 reply
random3 • today at 5:36 PM

Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.

➕ show 2 replies
dghlsakjg • today at 5:58 PM

How does this make you less reliant on third parties?

It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.

What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.

note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.

➕ show 3 replies