alt Hacker NewsReally interesting.
On the other hand, this only applies to a limited set of workloads (training is often memory bandwidth bound for example).
I also wonder how this affects power consumption.
Replies
Zagitta • today at 3:23 PM
It's also useful for prefill in inference, VLLM sees up to 30% boost depending on the exact kernels and prefill size
Training is compute bound.
If its memory bandwidth bound you need to raise your batch sizes to leverage more parallelism.