That benchmark doesn't mean what you think it means. (See the test description that you quoted.)

A score of 51% means that out of the total answers the model failed to answer correctly (out of 6000 questions in the benchmark), 51% were factually incorrect rather than non-attempted or uncertain.

This doesn't mean that Astra hallucinated 3060/6000 answers in the benchmark! (The hallucination rate could be 51% in that scenario only if Astra failed to answer a single question correctly.)

If the model failed to give a correct answer to only 100 out of the 6000 questions, but gave a hallucinated answer to 51 of those rather than expressing uncertainty, that would also give a hallucination rate of 51%.

It's a useful metric, but not what you're looking for here. The "Score" or "Accuracy" benchmarks are more what you're after.

(The frontier models still generate hallucinations on this hard set of problems, but it's not as bad as you think.)

Anyone that thinks that the hallucination rate is 59% has not actually used these models on a real project.

Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.

I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.

Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.

The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"

It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training

https://arxiv.org/abs/2404.01019

Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.

But then all the copy rights holders can simultaneously sue them.

And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?

Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.

With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").

It's telling that the companies prefer regulation over source-aware training.

EDIT: It's telling that the companies prefer regulation over source-aware training, which suggests the only additional regulation we need for now is mandating source-aware training?