alt Hacker News> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session
Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.
> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session
Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.
An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.