Dike Homme

[태그:] LLM hallucination

  • AI Hallucination, Explained — Why AI Makes Things Up and How to Deal With It

    Key Takeaway

    AI doesn’t lie. It was never designed to tell the truth in the first place.

    ChatGPT, Claude, Gemini — these tools are, at their core, an extreme version of your phone’s autocomplete. They predict the next most natural word based on patterns, not facts. Sometimes that prediction is right. Sometimes it produces something that sounds convincing but is completely fabricated. That’s what we call AI hallucination.

    Once you understand this, AI stops being scary. You don’t need to abandon it. You just need to know where it breaks.

    What Is an LLM, Really?

    The technology behind today’s AI chatbots is called an LLM (Large Language Model). The name sounds complex, but what it does is surprisingly simple.

    An LLM reads an enormous amount of text — news articles, research papers, Wikipedia, books, forum posts — and learns one thing: “Given this context, what word is most likely to come next?”

    Think of your phone’s autocomplete. Type “good morning” and it suggests “everyone” or “how are you.” Your phone doesn’t understand greetings. It just knows that “good morning” is statistically followed by certain words.

    An LLM is autocomplete on steroids. Trained on billions of sentences, it predicts the next natural word, appends it, then predicts again — hundreds or thousands of times in a row. That’s how you get those fluent, paragraph-length responses.

    It’s not thinking. It’s pattern-matching at scale.

    Why Does AI Hallucinate? — The Gap Between Pattern and Fact

    Ask an AI “What is the capital of France?” and it answers “Paris.” Correct.

    But did the AI know that Paris is the capital? No. In its training data, “the capital of France is” was overwhelmingly followed by “Paris.” It didn’t look up a fact — it continued a pattern.

    For frequently occurring information, this works remarkably well. The problem emerges when you ask about information that’s rare in the training data: a specific court case number, a niche research paper’s author, a regulation that was recently amended.

    When the pattern is thin, the AI doesn’t stop. It generates something plausible. A case number that looks real but doesn’t exist. An author name that sounds right but belongs to no one. A statute that was never enacted. That’s hallucination — confident fabrication where the AI fills gaps with plausible patterns instead of admitting uncertainty.

    Why Doesn’t AI Just Say “I Don’t Know”?

    This is the question everyone asks, and the answer is surprisingly mechanical.

    According to OpenAI’s 2025 paper “Why Language Models Hallucinate” (arXiv:2509.04664), the root cause lies in how AI models are scored during training:

    • Correct answer → 1 point
    • Wrong answer → 0 points
    • “I don’t know” → 0 points

    Imagine you’re taking a multiple-choice exam. You don’t know the answer to question 17. If you leave it blank, you get 0. If you guess, you have at least a 25% chance of getting it right. Any rational test-taker would guess.

    AI learned the same lesson. During training, guessing always scored equal to or better than abstaining. So the model developed a habit of answering confidently even when it shouldn’t. It’s not trying to deceive — it simply was never given an incentive to say “I’m not sure.”

    Real-World Consequences — This Isn’t Theoretical

    Fake Legal Citations in U.S. Courts

    In 2023, New York attorney Steven Schwartz submitted a legal brief containing six case citations generated by ChatGPT. Every single one was fabricated — fake case names, fake docket numbers, fake rulings. The court sanctioned Schwartz with a $5,000 fine, and the case became the first globally recognized incident of AI hallucination causing real legal consequences.

    It wasn’t an isolated event. In 2025, two attorneys in a Colorado defamation case involving MyPillow CEO Mike Lindell submitted AI-generated briefs containing over 20 fabricated citations, resulting in $3,000 fines each.

    As of 2026, the AI Hallucination Cases Database tracks over 1,300 documented incidents worldwide.

    A Global Pattern

    This is not just an American problem. In South Korea, a judge at Seoul Northern District Court identified fabricated Supreme Court case numbers in a plaintiff’s filing in 2025 and explicitly noted in the ruling that the citations were “AI-generated fakes”. Korean courts detected 10 such cases in 2023, 37 in 2024, and 73 in just the first half of 2025 — prompting the Supreme Court of Korea to establish a task force and add a “fake case number verification” feature to its judicial information portal (per Hankyung reporting).

    The Deeper Risk

    The courtroom cases are dramatic, but the everyday risk is more subtle. AI hallucinations sound exactly like correct answers. The tone doesn’t change when AI is wrong. Without verification, there’s no way to tell a real fact from a fabricated one — and people routinely make decisions based on AI outputs they never checked.

    AI Hallucination Rates Are Dropping Fast — But Context Matters

    The good news: hallucination rates have improved dramatically.

    According to the Vectara Hallucination Leaderboard, the lowest hallucination rate among leading models dropped from 21.8% in 2021 to 0.7% in 2025 — a 96% improvement in four years.

    But these benchmarks measure relatively simple tasks like summarization. In specialized domains, the picture is very different:

    • Medicine: A 2025 MedRxiv study tested AI on 300 clinical cases. Unmitigated hallucination rates reached 64–68%. Even with mitigation prompts, rates remained at 43–45%.
    • Law and Finance: Vectara’s 2026 enterprise benchmark showed hallucination rates above 10% for leading models including GPT-5 and Claude Sonnet 4.5.
    • Person-specific questions: OpenAI’s o3 hallucinated 33% of the time; o4-mini hit 48%.

    (Sources: Suprmind AI Hallucination Benchmarks 2026, All About AI Hallucination Report)

    The takeaway isn’t “don’t use AI.” It’s “know where AI is reliable and where it isn’t.”

    5 Practical Tips to Reduce AI Hallucination

    1. Tell the AI It’s Okay to Say “I Don’t Know”

    As we covered, AI has no built-in incentive to admit uncertainty. But when you explicitly instruct it — “If you’re not sure, say so instead of guessing” — the rate of confident fabrication drops noticeably. One sentence in your prompt can change the behavior.

    2. Ask for Sources

    Requesting citations reduces hallucination by roughly 40%, according to prompt engineering research. When AI is forced to back up its claims, it generates more carefully. And when it does hallucinate a source, you can catch it by checking whether the citation actually exists.

    3. Be Specific

    “Tell me about contract law” invites the AI to fill gaps with patterns. “What are the three requirements for breach of contract damages under Section 390 of the Korean Civil Code?” leaves far less room for fabrication. The narrower your question, the less the model needs to improvise.

    4. Ask AI to Verify Its Own Output

    After getting a response, follow up with: “Check your answer for any inaccuracies or unsupported claims.” AI models can catch their own logical inconsistencies and flag suspicious statements when prompted to self-review.

    For more advanced use, you can build self-verification into your workflow. For example, when using AI for legal research, include an instruction like: “After citing any statute or case, search to confirm it actually exists.” It’s an automated fact-check layer — simple to set up, significant in impact.

    5. Treat AI as a Draft Machine, Not an Answer Machine

    This is the most important mindset shift. AI’s strength is fast first drafts, not factual precision. Every AI output should go through human verification before being acted upon — especially numbers, proper nouns, legal citations, and medical information. If it would be catastrophic to get it wrong, check the original source.

    References