alt Hacker NewsHow should I update my simplistic understanding that decode is bw-bound with these results that show the B70 decoding faster than a 4090 (about 50% more bw)?
How should I update my simplistic understanding that decode is bw-bound with these results that show the B70 decoding faster than a 4090 (about 50% more bw)?
I doubt you'd get the same sort of result on a modern-ish MOE or dense model via a more standard inference engine like llama.cpp or VLLM. I don't think MLPerf is a reasonable benchmark at this point.
Edit: Here is a simple llama.cpp compare where the token gen results match the rule of thumb.
https://www.reddit.com/r/LocalLLaMA/comments/1st6lp6/nvidia_...