logoalt Hacker News

weitendorf • yesterday at 10:32 PM • 0 replies • view on HN

There's only a limited amount of hardware for training and serving models at scale, and which form an upper bound for what model size/inference can support.

A GB300 rack has 72x288GB=20TB VRAM. Do the math and 10T parameter models are just about the max size you can easily train and serve (at 1M context) without doing a lot of custom hardware development or limiting your scale.