alt Hacker NewsI'd argue the improvement has come not so much from fixing hallucinations, but more in getting LLMs to successfully run for longer, which lets them more effectively fix their own hallucinations. This is still an improvement, but at the cost of bigger models, with more tokens, running for longer.
I'd be interested to see benchmarks for this sort of thing plotted against cost, power usage, and elapsed time. If these are all sub-linear then that's great, but if some aren't that could be a real problem for LLMs in the long run.
There are too many variables. Autoregressive LLM architectures are not diffusion architectures are not JEPA architectures. We can't say right now how architectures are going to evolve or which one will become prominent even in the near future. There is still so much efficiency left on the table.
Plotting cost against power usage and elapsed time will tell you very little other than what the lay of the land is today. If you are truly concerned about the long term effects, then you need to account for long term evolution.