Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.

The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.