Cerebras is a large plate sized chip. It has 50GB of SRAM, and few hundred K simple cores that can access that SRAM really fast. I don't know semiconductors well, but I understand that the same manufacturing technique that makes this huge chip possible, on the flip-side limits inter-chip communcation bandwidth. In cerebras, it is 150 GB/s (compared to nvlink's 2TB/s or groq's similar).

One way large models are served on a bunch of cerebras chips is by essentially distributing layers' weights across chips. Few layers's weights per chip - as many as the KV cache + activations + weights will allow. You use pipelining to hide the latency of the inter-chip 150 GB/s link.

On GPUs, you amortize the cost of loading weights from HBM to SRAM across multiple users - thereby making it cheaper _per_ user. But here, there is no such amortization. The weights are already there. It is the activations that stream through.

You _could_ do batching/continuous batching, but that would just service more users at lower token/s each without any amortization of fixed cost, due to fixed cost (loading weights) being non-existent.