Why do LLMs hallucinate? AI hallucinations, explained
LLMs hallucinate because training and benchmarks reward a confident guess over 'I don't know'. Where hallucinations come from and how to cut them, in Python.

An AI hallucination is a fluent, confident answer from a language model that is false or unsupported. LLMs hallucinate because they are trained to produce plausible text, some facts can’t be learned from the training data at all, and most training and evaluation rewards a confident guess over “I don’t know”. The September 2025 paper Why Language Models Hallucinate by three OpenAI researchers and a Georgia Tech professor, frames it as a student guessing on an exam that gives no marks for leaving a question blank.
What counts as a hallucination?
The term covers two kinds of error. The Kalai et al. paper, following earlier taxonomies, separates:
- Intrinsic hallucinations, which contradict the prompt itself. Their example: asked “How many Ds are in DEEPSEEK?”, DeepSeek-V3 answered “2” or “3” across ten trials, and other models went as high as “6” and “7”.
- Extrinsic hallucinations, which contradict the training data or the world. Asked for the title of one author’s PhD dissertation, three popular models each gave a different, confident title and year, and none was right.
What makes them dangerous is the register. A hallucinated citation looks exactly like a real one. In the paper’s birthday example, a model asked for a date “If you know” gave three different specific dates on three attempts, all wrong. As the authors put it, bluffs are specific, such as “September 30” rather than “Sometime in autumn”.
Why do LLMs hallucinate?
There are three causes, and they stack.
1. Pretraining can’t learn every fact. A base model, the output of pretraining, learns the distribution of text. Kalai et al. connect generation to a binary question, “is this output valid?”, and show that whatever makes that classification hard also produces generation errors. Arbitrary facts with no pattern are the extreme case. Their result: after pretraining, the hallucination rate on such facts should be at least the fraction of them that appear exactly once in the training data. In their example, if 20% of birthday facts appear once, expect the base model to get at least 20% of birthdays wrong. The paper’s contrast: models “seldom err” on Einstein’s birthday, which is referenced many times, while a less famous person’s may appear once, in an obituary, and the model has no way to tell it apart from the 364 wrong dates.
2. The model doesn’t know what it doesn’t know, or doesn’t act on it. Anthropic’s interpretability work on Claude, Tracing the thoughts of a large language model (March 2025), found that refusing to answer is the default circuit, and a “known entity” feature switches it off for names the model recognizes, such as Michael Jordan. Hallucinations happen when that switch misfires: the model recognizes a name but knows nothing else about the person, so the “can’t answer” default is suppressed and it fills in something plausible.
3. Post-training and benchmarks reward guessing. This is the paper’s main argument. Most benchmarks grade answers right or wrong, with zero credit for “I don’t know”. Under that scoring, guessing always has a higher expected score than abstaining, so a model optimized for leaderboards learns to guess. The authors put it bluntly: models are always in “test-taking” mode.
Why does guessing win under accuracy scoring?
The paper’s proposed fix is to state a confidence target in the instructions, “Answer only if you are > t confident”, and to grade wrong answers at minus t/(1 - t) points, correct answers at 1 and “I don’t know” at 0. Then guessing pays only when your confidence is above t. This script scores three policies, always guess, abstain below 50% confidence, and abstain below 90%, under three graders.
# guessing.py: why a model graded on accuracy learns to guess.
# Scores three answer policies under binary grading and under the confidence-target
# grading proposed by Kalai et al. (2025). Python 3, stdlib only.
import random
random.seed(1)
# 10,000 questions. For each, the model has a confidence c that its best answer is
# right and, if it is calibrated, it is right with probability c.
conf = [random.betavariate(2, 2) for _ in range(10_000)]
correct = [random.random() < c for c in conf] # fixed, so every policy faces the same luck
def score(abstain_below, t):
penalty = t / (1 - t) # wrong costs t/(1-t), "I don't know" costs 0
s = wrong = idk = 0
for c, ok in zip(conf, correct):
if c < abstain_below:
idk += 1
elif ok:
s += 1
else:
s -= penalty; wrong += 1
return s / len(conf), wrong, idk
policies = {"always guess": 0.0, "IDK if c < 0.5": 0.5, "IDK if c < 0.9": 0.9}
graders = {"binary (t=0)": 0.0, "t=0.5, penalty 1": 0.5, "t=0.9, penalty 9": 0.9}
print(f"{'':<17}" + "".join(f"{g:>19}" for g in graders))
for name, cut in policies.items():
cells = [score(cut, t) for t in graders.values()]
print(f"{name:<17}" + "".join(f"{s:>+19.3f}" for s, _, _ in cells),
f" (wrong {cells[0][1]}, IDK {cells[0][2]})")
Output (Python 3.13.11):
binary (t=0) t=0.5, penalty 1 t=0.9, penalty 9
always guess +0.492 -0.016 -4.078 (wrong 5078, IDK 0)
IDK if c < 0.5 +0.339 +0.180 -1.091 (wrong 1589, IDK 5024)
IDK if c < 0.9 +0.026 +0.025 +0.013 (wrong 15, IDK 9723)
Read down each column. Under binary accuracy, “always guess” wins with +0.492, despite giving 5,078 wrong answers out of 10,000. The policy that says “I don’t know” when unsure produces a third as many wrong answers and scores lower. That’s the incentive every accuracy leaderboard sends.
Under the t = 0.5 grader the best policy is to abstain below 50%, and under t = 0.9 it’s to abstain below 90%. The best behavior matches the stated threshold every time, which is the paper’s point: if the grading says what confidence it wants, a calibrated model can be honest without being punished for it. The simulation assumes a perfectly calibrated model, which real ones aren’t, but the incentive holds either way.
The paper also notes this isn’t new to machine learning. Exams such as India’s JEE, NEET and GATE, and the SAT and GRE in earlier years, penalize wrong answers, and the instructions say so, so test takers know when a guess is worth making.
How do you reduce hallucinations in practice?
You can’t remove them from the model, but a system can make them rare and catchable.
- Ground the answer. Put the facts in the context with retrieval or a tool call, and instruct the model to answer only from that material. Most enterprise hallucinations are questions the weights were never going to know.
- Allow abstention, explicitly. Say in the prompt that “I don’t know” or “not in the documents” is an acceptable answer, and don’t penalize it in your own evals. Your eval suite is a benchmark too; if it scores only accuracy, you’re training your prompt toward guessing.
- Ask for citations and check them. Require the answer to quote or point to the source span, then verify mechanically that the quote exists. A citation you didn’t check is decoration.
- Verify what can be verified. Run the code, execute the SQL, recompute the number, validate against a schema. A check costs less than a wrong answer in production.
Turning the temperature down isn’t on that list. A lower temperature makes the model repeat its most likely answer, and its most likely answer can be the hallucination.
What I would do
Design every LLM feature on the assumption that some answers will be confidently wrong, and decide up front what happens when one is. For a draft email, a human reads it. For a number going into a report, a deterministic check recomputes it. For a support answer, the reply links its source so the reader can verify.
From a data-platform seat, the line I’d circle is the singleton result. Facts that appear once in the training data are the ones the model gets wrong, and your company’s internal facts appear zero times. That’s the strongest argument I know for grounding answers in your own data rather than trusting what a model “knows” about your domain.
rSearch, the search product I built, follows the same principle on the open web: DeepSeek R1 reasons over results retrieved through Serper instead of answering from its weights alone.
FAQ
Can AI hallucinations be completely eliminated?
No, not for a model that answers open-ended questions. The Kalai et al. analysis gives a lower bound on errors for facts with no learnable pattern. What you can do is make a system abstain more, ground answers in sources and check outputs, which brings the rate down a long way.
Do newer models hallucinate less?
Not to zero. The Kalai et al. paper says hallucinations are still present in the latest models. The singleton argument predicts where they persist: rare facts, facts after the training cutoff and facts that were never public. Well-known facts, seen many times in training, are where models rarely slip.
Does RAG stop hallucinations?
It reduces them a lot when retrieval finds the right passage. If retrieval returns the wrong passage or nothing, the model may still answer from memory, so tell it to abstain when the context doesn’t contain the answer and measure how often it does.
Why is it called a hallucination?
It’s a borrowed word for a model producing content that isn’t grounded in its input or in fact. The Kalai et al. paper notes it “differs fundamentally from the human perceptual experience”; “confabulation” is arguably the better term, but “hallucination” stuck.
Sources
- Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate, September 4, 2025
- Anthropic, Tracing the thoughts of a large language model, March 27, 2025