alt Hacker NewsOne problem we’re going to have with AI hardware is coming up with a standard set of specifications that are comparable. I don’t really care about the CPU GHz and the memory bandwidth, at least not directly. What I really want to know is how many tokens per second this will deliver, but that also depends on the model. We need a standard metric for that. Perhaps we agree on a specific open weight model (e.g. GLM 5.3 Flash or Qwen vWhatever) and then measure TPS on the hardware of interest.
Replies
> I don’t really care about the CPU GHz and the memory bandwidth ... What I really want to know is how many tokens per second this will deliver
There is a fairly direct link between the two numbers. You can predict the latter from former reasonably well
I always heard of GB/s as most important number ... AI told me their 8800 MT/s on 8 Byte, 12 DDR5-Channels means 845 GB/s. A Nvidia RTX 4090 has 1008 GB/s. Nvidia B200 has 8000 GB/s. Is this the right way of looking at it?
[flagged]
I feel like even TPS is becoming less of a good metric as we're seeing certain models handle similar problems while burning far fewer tokens.