There is 1 G in CAT: Living with AI Hallucinations

Plato's Allegory of the Cave: an early work on the issues of illusory data. (Art by Jan Saenredam via wikimedia)

LLMs and other generative AI products have transformed how organizations interact with data, but they have come with a new kind of challenge: hallucinations. Simply put, hallucinations are moments when an AI model generates convincing but false information, and uses the false information for follow-on tasks or conveys it to an end user. These hallucinations can range from the obvious and possibly humorous, such as when the LLM states "there is 1 G in Cat," to the potentially catastrophic, such as when the LLM hallucinates an entire set of cases in a legal brief.

For enterprises relying on AI to make decisions, hallucinations are more than an inconvenience, they are a risk, since unverified AI outputs can lead to bad decisions, compliance violations, or reputational harm. There can be significant costs associated with rework to repair the damage caused by hallucinations, which can undermine the entire value proposition of using LLMs in the first place.

Originally, it was assumed that hallucinations occurred because a given LLM lacked context, had incomplete data, or was fed from sources of questionable quality. However, new research from OpenAI and Georgia Tech postulates that hallucinations are a byproduct of both the pre-training statistics behind LLM response generation and the way LLMs are trained. If true, this means that hallucinations are not likely to go away anytime soon, even as models improve, since they are inherent in the processes that all major modern LLMs use. So how does this happen, and what can enterprises do about it?

The Pre-training Math Problem

During pre-training, a large language model is asked to predict the next token (word, subword, etc.) given its context. The model sees billions of text sequences and tries to minimize its prediction error. Suppose the training body has many cases where both true and false statements appear in similar contexts (e.g., “Einstein was born in Germany” vs. “Einstein was born in Austria”). If the model cannot reliably distinguish between them it still has to assign them probabilities, since the "loss function" penalizes the model for failing to assign probability to any observed token in the data. It must do this even if the two options are contradictory so both "Einstein was born in Germany" and "Einstein was born in Austria" get assigned weights, even though Einstein could only have been born in one place. That means the model gets pushed to “hedge its bets” by assigning nonzero probability to both true and false tokens.

Over time, this creates statistical pressure, since even if the model "knows" (in the loosest sense of the word) that it doesn’t have enough information to choose correctly, the loss function punishes it for abstaining. The model learns to confidently output plausible but false statements when faced with ambiguity. As a final note for the intellectually curious, I will add that Einstein was born in Germany (specifically the Kingdom of Württemberg) and I encourage readers to ask their favorite LLM what it thinks about that.

The Trouble With Training

Because LLMs are trained to answer questions, many popular benchmarks used to evaluate LLM performance penalize "I Don't Know," responses. This results in the model The researchers provide an analogy to students taking standardized tests: "When uncertain, students may guess on multiple-choice exams and even bluff on written exams, submitting plausible answers in which they have little confidence. Language models are evaluated by similar tests." This, according to the researchers, increases the pressures on models to give answers even in the face of uncertainty.

The Takeaway

The upshot of this is that, even with perfect data and a known response set, hallucinations will still occur. This means that organizations need better data controls to deal with the inevitability of hallucinations if they want to rely on LLMs for anything but the most trivial workflows. So, how can an organization mitigate the risks of hallucinations?

Audit Trails for Every Query: Require complete auditability of who accessed what data, when, and how it was applied to an AI workflow. This creates traceable evidence behind every AI-driven insight.

Granular Access and Classification Controls: Use dynamic rules to ensure that models (and the users behind them) only receive the right data, based on classification, role, and ownership.

Context-Rich Data Pipelines: Document data across sources, and provide AI systems with well-labeled, policy-aware data instead of isolated silos, reducing the need for the system to "guess."

Contact us at info@infobastion.com for a free consultation and demo to see how Info Bastion's unified data platform, Bastilon, can assist with these tasks.