logoalt Hacker News

random3 • today at 5:36 PM • 2 replies • view on HN

Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.


Replies

lrvick • today at 5:47 PM

Really depends on what GPUs you go with but I hate proprietary software so I went with AMD AI PRO r9700 cards which you can still pick up used on ebay for $1500 used on auctions. Started with 1 then expanded over time as I won auctions. Though on a lower budget my $400 16gb 6800 XT can run Qwen 3.8 27b which I could absolutely get by with at significantly reduced speed.

My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.

Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.

Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.

Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.

➕ show 1 reply
vablings • today at 5:42 PM

There are tons of resources out there on how to run a local setup. Once you have something that's somewhat functional you can then have the system self optimize your software stack to suit your needs.

As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.