Check out Taalas, recently acquired by AMD. They printed a model directly to silicon and it generates 16,000 tokens per second. You can even try a demo here:
Whats amazing is that this is around 5x faster than the top end Cerebras wafer-scale silicon, at much less power/cooling cost per token. Of course Cerebras is a general purpose design, but perhaps Taalas could license CBRS wafer-scale patents to create a single monster chip that held either a single massive trillion parameter-scale model, or could an MoE design with a half dozen smaller specialized models on the same wafer with a shared SRAM pool.
I wonder if cooling such a beast, with very non-uniform hot spot distribution, would be too complicated to bother with though.
Whats amazing is that this is around 5x faster than the top end Cerebras wafer-scale silicon, at much less power/cooling cost per token. Of course Cerebras is a general purpose design, but perhaps Taalas could license CBRS wafer-scale patents to create a single monster chip that held either a single massive trillion parameter-scale model, or could an MoE design with a half dozen smaller specialized models on the same wafer with a shared SRAM pool.
I wonder if cooling such a beast, with very non-uniform hot spot distribution, would be too complicated to bother with though.