Well, Taalas has that kind of technology, but the chip they demoed is probably 20-100 times smaller than necessary since it's only an 8b model.

But let's say they could someday scale that up to a much larger model, 72 large chips per wafer and each chip can do 1000 LLM requests at once (Vera Rubin?). So it's roughly the equivalent of an NVL72 rack.

You might be able to serve something like 50000-60000 requests at once. So I think it's more like handling a small city's worth of customers per wafer than the world if you had that.

I believe in less than 5 years we will get to that, but the model size and/or number of agents is going to keep going up also.