alt Hacker NewsHow?
edit: now I read the article fully, seems like they utilize some very effective MTP algorithm. and somehow the quality is still decent enough.
though, I doubt that the quality really only drip a bit like they claimed. maybe for the benchmarks, but for general uses the heavily quantized models very often so worse result.
Replies
lostmsu • today at 4:43 PM
They say they are using https://github.com/tile-ai/TileRT
- persistent CUDA kernel
- tiled processing with overlapping read/writes
- model designed with specific constraints in mind
➕ show 1 reply
i wonder if it will be possible to hardcode a model with some kind of MTP-adjacent algorithm to use a smaller portion of it to generate most of the tokens but route to the real experts every once in a while to steer it towards good thinking directions. (Perhaps this is done only when it's generating its thinking block, and the training takes it into account)
Could result in very high efficiency and still good intelligence without having to resort to fundamental adjustments like going to a diffusion LLM