logoalt Hacker News

cookiengineer • today at 1:10 AM • 0 replies • view on HN

It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).

Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.

[1] https://github.com/cookiengineer/qwen-reasoning-trainer