logoalt Hacker News

arcanemachinertoday at 5:39 PM0 repliesview on HN

Dollars to donuts, they are speculating, and not privy to inside information on the topic.

However, I believe that runtime model quantization is possible with some publicly-available inference engines (e.g. vLLM), so its not beyond belief that the closed labs do quantize at runtime, either to allocate compute, or to nudge users towards a preferred model (e.g. make the incumbent model dumber to push people to use the latest-and-greatest model, or vice versa to ease the load on the latest model, which is typically larger than the old one).