LLM Hallucinations: Why They Happen and How to Reduce Them

A hallucination is output that is fluent and confident but false or unsupported: an invented citation, a nonexistent function, a wrong date stated as fact. Hallucinations are not occasional glitches that will disappear; they follow from how language models work, and any system built on them needs to be designed with that in mind.

Why models hallucinate

A language model is trained to predict plausible next text. It does not look facts up in a database; it generates what is likely given its training and the prompt. This leads to several causes:

• Plausibility over truth. The most probable continuation of "The paper by Smith et al. showed" is a confident-sounding claim, whether or not such a paper exists.
• Gaps in training data. For rare topics, recent events, or private information, the model has little to draw on, yet still produces an answer.
• Pressure to answer. Models tuned to be helpful tend to answer rather than say "I don't know."
• Missing or irrelevant context. If the needed information is not in the prompt, the model fills the gap.
• Long-context and reasoning errors. Mistakes in multi-step reasoning accumulate.
• Ambiguous questions. The model guesses the intent instead of asking.

Importantly, confidence in the wording is not a reliable signal of accuracy.

The security angle: invented packages

When asked to write code, models sometimes recommend libraries that do not exist. Research on code-generating models has measured that a meaningful fraction of recommended package names can be invented. Attackers can exploit this by registering a package under a commonly hallucinated name and filling it with malicious code, so a developer who installs what the model suggested runs the attacker's software. This is sometimes called "slopsquatting."

Defenses:

• Verify every unfamiliar package: check that it exists, its age, maintainers, download history and source repository before installing.
• Use lockfiles and an internal package registry or allowlist.
• Run dependency scanning in CI (secure pipelines).
• Do not paste model-generated install commands into production environments without review.

Mitigations that work

1. Ground the model in sources

Provide the relevant documents and instruct the model to answer only from them and to cite them. This is the main purpose of retrieval-augmented generation. Grounding greatly reduces, but does not eliminate, fabrication, because models can still misread or embellish the sources.

2. Require citations and check them

Ask for a quotation or passage ID for each claim, then verify in code that the cited passage exists and supports the statement. A claim with no valid citation can be removed or flagged.

3. Allow "I don't know"

Give explicit permission and instructions to decline when the sources are insufficient, and evaluate that behavior. Reward correct refusals in your tests, not only correct answers.

4. Constrain the output

Use structured output with enumerated values where possible. If a field must be one of five categories, a schema prevents inventing a sixth. For factual lookups, have the model call a tool or database instead of recalling from memory.

5. Verify with tools

For calculations, dates, code behavior and facts that can be checked, call a calculator, run the code, or query the authoritative system. Use the model to decide what to check, not to be the source of truth.

6. Use a second pass

A separate step can compare an answer to the sources and flag unsupported statements. This catches many errors but is itself a model and can be wrong, so combine it with other controls.

7. Keep a human in the loop for high stakes

For medical, legal, financial or safety-relevant output, have a qualified person review before it is used. Design the interface to show sources next to claims so review is fast.

Measuring hallucination

You cannot manage what you do not measure.

• Build a test set of questions with known answers, including some that cannot be answered from the provided material.
• Score faithfulness (is every claim supported by the sources?) and correct refusal separately from plain accuracy.
• Sample live outputs for review and feed new failures into the test set (evaluation practice).

What not to rely on

• Telling the model "do not hallucinate" in the prompt. It helps little by itself.
• The model's self-reported confidence.
• Assuming a newer or larger model has eliminated the problem.

Communicating limits to users

Be honest in the interface: show sources, label generated content, and make it easy to report mistakes. Users who understand that output needs checking are safer than users who assume it is authoritative. Misinformation is also a named risk category in the OWASP LLM Top 10.