alt Hacker NewsSo this is kind of fascinating. The main hardware costs here seem to be:
- 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each
- 2x 100G Ethernet controllers for ~$500 each
So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapter or something to even get them to work.
I don't know how hot a 100GbE copper NIC runs but, from experience, 10GbE NICs have been basically giant heatsinks, basically. So fiber might be advisable and I expect short fiber cables here probably aren't cost-prohibitive given everything else.
As an aside, if you are using Ethernet for clustering and you're clustering 2 devices, in an ideal world you'd be using simplex Ethernet but that's not an option here.
I wonder if the author considered USB 4.0 for clustering? I ask because I know people who have clustered Mac Studios over TB5 and that bandwidth is up to 120Gbps. The version of USB4 on the Ryzen AI 395 seems to be 40Gbps, which isn't that far off 8GB/s over PCI-e 4.0 x4.
But the limiting factor with Strix Halo (and DGX Spark for that matter) is memory bandwidth, both under 300GB/s. The obvious comparison is to the Mac Studio. Unfortunately the largest spec they currently sell is 96GB. It had been as high as 512GB. And 96GB is $6700+ but you're also getting way better performance AFAICT eg [1]. The M3 Ultra has ~900GB/s memory bandwidth.
You can alternatively buy a Macbook Pro with M5 Max and 128GB of RAM (now $8000, was $5500-6000 a few days ago) but that tops out at ~600GB/s, which is still double these mini AI boxes.
Oh and if you don't want to go the way of these Framework motherboards, you can buy a whole 128GB Strix Halo PC for $3k or less.
I think the main point here though is we're only a few years away from running 300B+ (or even 1T+) param models at useful speeds on enthusiast hardware.
[1]: https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can...
Replies
I ran Ms-01s with 100GBE, copper DACs in my kubernetes cluster. Killed the NVME drives in that tiny box. I'd bet the same issue doing this with FW. And I wasn't even pushing 100GBE very hard at all, it was mostly for fun.
AI + 100GBE (under load) + tiny box = unreliable and eead very quickly.
No reason to use fiber on short runs like that. DAC cables are cheap and better in pretty much every way over short distances. You're probably thinking of RJ-45 NICs and SFP modules which are known to run pretty hot.