alt Hacker NewsI don’t think that’s what’s going on. I notice flaws on day one of model releases. But I also notice improvements if the model is truly more advanced than what I’m used to. Then over time the same questions or tasks return worse results.
What is actually stopping these model companies from running a model at full capacity on release then once its name rings out, start serving users quantized garbage?
Replies
It's called hedonic adaptation.
> What is actually stopping these model companies
You can say this about any company in the world, selling anything.
It's trivially measurable, and there are people running the same benchmark on the leading models every day and measuring if they degrade. Spoiler: they don't.
But you can always say "the conspiracy goes higher", and that the companies know about these daily benchmarks and are routing them to "quality" envs.
> What is actually stopping these model companies from running a model at full capacity on release then once its name rings out, start serving users quantized garbage?
...I mean, if they were actually doing this despite saying that they don't—promising one product and delivering something else—I think that would be fraud, no?
And, maybe it's one thing to secretly defraud normies like us (although class action lawsuits do exist), but I don't think major enterprises or the US military would take too kindly to it.