Why do LLMs hallucinate?

A 2025 paper by Kalai, Nachum, Vempala and Zhang argues that training and evaluation reward guessing over admitting uncertainty, so models learn to produce a plausible answer rather than say they do not know. Gaps or conflicts in the source data make this worse.

Is hallucination the same as being out of date?

Not quite. An outdated answer repeats something that was once true. A hallucination states something that was never true. Both cause wrong answers about businesses, and both shrink when current facts are easy to retrieve.

Does web search stop hallucinations?

It reduces them. Retrieval-augmented generation grounds the answer in fetched documents, and the original RAG paper found more factual output than a model without retrieval. Models can still misread or mix sources, or guess when retrieval finds nothing.

What can I do if ChatGPT says something false about my company?

Find the prompts that trigger it, check what sources the engine cites, then publish or correct the facts on your own site and on the profiles engines draw from. Re-run the same prompts after crawlers revisit to see whether the answer changed.

AI search glossary

LLM hallucination: why AI models state false facts, and about your brand

An LLM hallucination is output from a large language model that contains false or misleading information presented as fact, such as an invented citation, a wrong price, or a feature a product does not have.

Also called: AI hallucination, confabulation, hallucinations in large language models

Why it matters for AI search

When an AI engine lacks clear, retrievable facts about your business, it may fill the gap with a plausible guess: an old price, a discontinued product, a competitor's feature, or a founding story that never happened. Buyers rarely check. Publishing plain, consistent, crawlable facts about what you sell, what it costs and who it is for gives the engine something to retrieve instead of something to invent.

What a hallucination looks like

Wikipedia's article defines an AI hallucination as a response containing false or misleading information presented as fact. Other names in use include confabulation. Documented cases show the range:

Invented legal cases. In Mata v. Avianca (2023), lawyers filed a brief citing six court cases ChatGPT had made up; the judge fined them $5,000.
Invented company policy. In Moffatt v. Air Canada (2024), the airline's support chatbot described a bereavement fare policy that did not exist. A Canadian tribunal ordered the airline to honor it and rejected the argument that the chatbot was responsible for its own statements.
Invented sources. Meta's Galactica science model (2022) cited papers that do not exist and was withdrawn within days.

The Air Canada case is the one businesses should remember: the false statement was about the company's own offer.

Why models hallucinate

A language model predicts likely text. When the right fact is missing, rare or contradicted in its data, the likeliest text can still be wrong. Kalai, Nachum, Vempala and Zhang (2025) add an incentive argument: common benchmarks score a confident guess better than "I don't know", so models become good test-takers that guess. They propose changing how existing benchmarks are scored so honest uncertainty is not penalized.

Retrieval-augmented generation helps by putting fetched documents in front of the model. It does not fix everything: if retrieval returns nothing about you, or returns a stale third-party page, the model writes from that.

Hallucinations about your brand

For a business, the common failure modes are:

What the AI says	Usual cause
A wrong or old price	No current price in crawlable text; an old review page ranks instead
A feature you do not offer	Confusion with a competitor or a similarly named company
"No information available" or a guess	Search crawlers blocked, or facts only inside images or scripts
Wrong founder, location or founding year	Inconsistent profiles across your site, directories and Wikidata

How to reduce them

Ask the engines. Run the questions buyers ask about you in ChatGPT, Claude, Gemini and Perplexity, and record exactly what is wrong. This is the core of AI brand monitoring.
Publish the facts plainly. Price, plans, availability, who it is for, location: in HTML text on pages that are allowed in robots.txt.
Make your identity unambiguous. Use consistent names and profiles everywhere, backed by Organization markup, which is the work of entity SEO.
Fix the sources engines cite. If an answer cites an outdated directory or review page, update that page where you can.
Re-check after recrawls. Corrections only show up once crawlers have fetched the new pages.

Questions

Why do LLMs hallucinate?: A 2025 paper by Kalai, Nachum, Vempala and Zhang argues that training and evaluation reward guessing over admitting uncertainty, so models learn to produce a plausible answer rather than say they do not know. Gaps or conflicts in the source data make this worse.
Is hallucination the same as being out of date?: Not quite. An outdated answer repeats something that was once true. A hallucination states something that was never true. Both cause wrong answers about businesses, and both shrink when current facts are easy to retrieve.
Does web search stop hallucinations?: It reduces them. Retrieval-augmented generation grounds the answer in fetched documents, and the original RAG paper found more factual output than a model without retrieval. Models can still misread or mix sources, or guess when retrieval finds nothing.
What can I do if ChatGPT says something false about my company?: Find the prompts that trigger it, check what sources the engine cites, then publish or correct the facts on your own site and on the profiles engines draw from. Re-run the same prompts after crawlers revisit to see whether the answer changed.