logoalt Hacker News

kakugawa • today at 8:28 PM • 3 replies • view on HN

Yeah, a good balance would be a local reasoning model paired w/ a knowledge pack + web-based searches. You want your obscure coding language? Provide a pre-vetted knowledge pack locally that it knows how to use. Same with any other domain-specific knowledge.


Replies

Tade0 • today at 9:05 PM

For that, Qwen 3.8 Flash Next is split into two shards, one with the model, the other with the n-gram table, which functionally holds its world knowledge.

Personally I recommend ISTA-DASLab's version, pruned to half the number of experts to specialise in coding:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

I didn't think my somewhat dated RX7700S would be supported in Strata, but it is and I'm getting 120tps+ prefill and 20tps+ generation with it, which is already practical for my purposes.

wolvoleo • today at 8:30 PM

Yes especially now that local context isn't so costly anymore. With llama3 it would kill my vram just to have a few thousand tokens. With recent models it's hardly an impact anymore. Not sure how they manage that but it's great. It still takes more processing but I also run an older card that is not very good at prefill.

jeffyaw • today at 9:01 PM

That’s what I do with typed, there’s a knowledgebase that loads with an embedding model for code related lookups.