alt Hacker NewsThat's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
Replies
The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.