logoalt Hacker News

syntaxing • today at 7:54 PM • 6 replies • view on HN

I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.


Replies

Aurornis • today at 8:12 PM

It's less than 24 hours old. The inference engines haven't been optimized for it yet.

flatline • today at 8:39 PM

I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.

➕ show 1 reply
vineyardmike • today at 7:57 PM

According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.

➕ show 1 reply
baby • today at 8:43 PM

Can you autoresearch optimize it?

woadwarrior01 • today at 7:59 PM

The NV in NVFP4 stands for NVIDIA. :)

plasticchris • today at 7:56 PM

It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.