If you’ve used AI for any real work, you’ve probably caught it confidently stating something that’s just wrong. This is called hallucination — when an AI generates information that sounds plausible but is entirely fabricated. This AI hallucination comparison breaks down which model gets it wrong the most.
But which model actually hallucinates the most? GPT, Claude, or Gemini? Gut feelings don’t count here. You need data.
I compiled the major benchmark results published between 2025 and 2026 to answer this question as objectively as current evidence allows.
TL;DR (2026 hallucination rates, Vectara benchmark): GPT-5.4-nano 3.1%, Gemini 2.5 Flash-Lite 3.3%, Grok 3 5.8%, Claude Haiku 4.5 9.8%, Claude Opus 4.6 12.2%, Gemini 3 Pro 13.6%. Lower = better. Data last updated April 20, 2026.
Benchmark-by-Benchmark AI Hallucination Comparison
Comparing AI hallucination requires understanding how it was measured. The same model can score vastly differently depending on the benchmark. Here are the two most widely cited benchmarks and what they found.
1. Vectara — Document Summarization Test
The most widely referenced hallucination leaderboard in the industry. The methodology is straightforward: give the AI a document, ask it to summarize, then measure how much content it fabricated that wasn’t in the original.
Until April 2025, the test used about 1,000 short documents. In February 2026, Vectara overhauled the dataset: 7,700+ long articles (up to 32K tokens) spanning legal, medical, financial, and technical domains. After this upgrade, hallucination rates across all models jumped 3–10x.
Legacy Dataset (April 2025, ~1,000 short documents) — Ranked by hallucination rate (highest first)
| Model | Hallucination Rate |
|---|---|
| Claude 3.7 Sonnet | 4.4% |
| GPT-4.1 | 2.0% |
| o3-mini-high (OpenAI) | 0.8% |
| Gemini 2.0 Pro-Exp | 0.8% |
| Gemini 2.0 Flash | 0.7% |
New Dataset (Feb 2026, 7,700 long documents) — Ranked by hallucination rate (highest first)
| Model | Hallucination Rate (lower = better) |
|---|---|
| GPT-5.4-nano (2026-03-17) | 3.1% |
| Gemini 2.5 Flash-Lite | 3.3% |
| o3-mini-high (OpenAI) | 4.8% |
| GPT-4.1 | 5.6% |
| Grok 3 (xAI) | 5.8% |
| Gemini 2.5 Pro | 7.0% |
| Claude Haiku 4.5 | 9.8% |
| Claude Sonnet 4.6 | 10.6% |
| Gemini 3.1 Pro Preview | 10.4% |
| GPT-5.2-high | 10.8% |
| Claude Opus 4.6 | 12.2% |
| Gemini 3 Pro | 13.6% |
On the legacy dataset, most models stayed under 5%. On the new dataset, every state-of-the-art reasoning model exceeded 10%. Longer, more complex documents make every AI fabricate more.
Sources: Vectara Hallucination Leaderboard, Vectara Blog — Introducing the Next Generation, Suprmind AI Hallucination Rates & Benchmarks
2. AA-Omniscience — The Test Where “I Don’t Know” Is a Valid Answer
This benchmark tests 6,000 questions across 42 topics in 6 domains (business, humanities, health, law, software engineering, science/math), measuring both knowledge and hallucination simultaneously.
What makes it unique: saying “I don’t know” incurs no penalty — only wrong answers are penalized. This directly measures the trait that matters most in real-world AI use: the ability to admit uncertainty rather than fabricate an answer.
The Omniscience Index ranges from -100 to 100 (higher is better). A model that frequently gives wrong answers can go negative. Ranked by hallucination rate (highest first):
| Model | Accuracy | Hallucination Rate | Omniscience Index |
|---|---|---|---|
| Gemini 3 Flash | 54.0% | 91% | — |
| Gemini 3 Pro | 55.9% | 88% | 16 |
| GPT-5.1 (high) | 37.6% | 81% | — |
| GPT-5.2 (xhigh) | 43.8% | ~78% | — |
| Claude Opus 4.5 | 45.7% | 58% | — |
| Gemini 3.1 Pro Preview | 55.3% | 50% | 33 |
| Claude Sonnet 4.6 | — | ~38% | — |
| Claude 4.5 Haiku | — | 25% | — |
| Claude Opus 4.6 | 46.4% | — | 14 |
Why High Accuracy and High Hallucination Can Coexist
You might notice something odd: how can a model have both high accuracy and high hallucination? Shouldn’t high accuracy mean low hallucination?
The key is the third option: refusal. When an AI receives a question, it does one of three things: answer correctly, answer incorrectly, or refuse to answer. These three add up to 100%.
Think of it like an exam where you can skip questions with no penalty.
- Gemini 3 Flash attempted all 100 questions. Got 54 right, got 46 wrong. Accuracy: 54%, hallucination: 91%. It answers a lot correctly, but it also guesses on questions it doesn’t know — and gets many wrong.
- Claude 4.5 Haiku only attempted questions it was confident about and skipped the rest. Fewer correct answers, but far fewer wrong ones. Hallucination: 25%.
It’s the difference between a student who attempts every question and one who only answers what they’re sure about. Accuracy is “correct answers / total questions,” while hallucination rate is closer to “wrong answers / attempted questions.” That’s why both can be high simultaneously.
Why this matters in practice: GPT-5.1 has 37.6% accuracy with 81% hallucination. It answers most questions, but a large portion of those answers are wrong. Claude models, by contrast, have lower accuracy but far lower hallucination rates among the questions they do answer. The “refuse when uncertain” strategy is genuinely working.
Sources: Artificial Analysis — AA-Omniscience, Suprmind AI Hallucination Rates & Benchmarks
So Which AI Is Most Accurate?
Synthesizing both benchmarks in this AI hallucination comparison, the answer depends on the task. No single model wins across all scenarios.
Document-Based Tasks (Summarization, Analysis, Research): Gemini Leads
For tasks like “summarize this” or “analyze this contract,” Gemini models consistently show the lowest hallucination rates. Gemini 2.0 Flash ranked #1 on the Vectara legacy dataset at 0.7%, and Gemini 2.5 Flash-Lite ranked #1 on the new dataset at 3.3%.
This makes sense. Google is a search company. Accurately extracting information from given documents is what Google has been doing for decades. That DNA carries over to Gemini.
Knowing What It Doesn’t Know: Claude’s Strength
On AA-Omniscience, Claude 4.5 Haiku achieved a 25% hallucination rate — the lowest among all three providers. Due to Anthropic’s Constitutional AI training approach, Claude tends to refuse rather than guess when uncertain. If your priority is “a wrong answer is worse than no answer,” Claude is the relatively safer choice.
Where Does GPT Stand?
GPT occupies the middle ground across both benchmarks. On the Vectara new dataset, GPT-5.2-high scored 10.8% — nearly identical to Claude Sonnet 4.6 at 10.6%. On AA-Omniscience, GPT-5.1’s 81% hallucination rate is lower than Gemini 3 Pro (88%) but significantly higher than the Claude family (25–58%).
However, GPT-4.1 (a non-reasoning model) scored a solid 5.6% on Vectara. Within the GPT family itself, the gap between reasoning models (GPT-5 series) and non-reasoning models (GPT-4.1) is substantial. That’s why a one-line verdict on “GPT” as a whole doesn’t work.
The Most Interesting Finding: The Reasoning Model Paradox
One consistent pattern has emerged from 2025–2026 hallucination research:
“Models that think harder fabricate more.”
On the Vectara new dataset, every reasoning/thinking model — GPT-5, Claude Sonnet 4.6, Grok-4, DeepSeek-R1 — exceeded 10% hallucination. Meanwhile, the non-reasoning Gemini 2.5 Flash-Lite sat at just 3.3%.
For readers searching specifically for Grok 4 hallucination rate vs Claude, GPT, and Gemini in 2026: on the same Vectara 2026 dataset, Grok-4 sits in the same 10%+ band as GPT-5 and Claude Sonnet 4.6, while the non-reasoning Gemini Flash-Lite variant remains the outlier at 3.3%. The full Vectara Hallucination Leaderboard (2026, Claude vs GPT vs Gemini vs Grok) is linked at the bottom of this article and updated as new models enter the benchmark.
Why does this happen?
Reasoning models take “thinking time” before answering. For math problems and logic puzzles, this is a major advantage. But for document summarization, it backfires. Instead of faithfully reproducing what’s in the source, the model reasons: “Given this context, there should also be…” and adds information that doesn’t exist in the original.
It’s like assigning meeting minutes to a diligent new hire who decides, “Given the discussion, the conclusion must have been X” — and writes down a conclusion that was never actually reached. The smarter they are, the more confidently they make this mistake.
Practical takeaway: For tasks where fidelity to source material matters — summarization, analysis, research — a non-reasoning model (Flash-tier, etc.) may actually be safer than the latest reasoning model. For tasks requiring complex reasoning (math, coding, strategic planning), reasoning models still dominate. Match the model to the task.
How to Interpret Benchmark Numbers
Different Benchmarks Measure Different Things
Even in this AI hallucination comparison, the same model’s hallucination rate can be 3% in one and 50% in another. This isn’t the model being inconsistent — the benchmarks are measuring fundamentally different things.
| Benchmark | What It Measures | Analogy |
|---|---|---|
| Vectara | Summarization fidelity | “Open-book exam” |
| AA-Omniscience | Knowledge + self-awareness | “Exam where you can skip questions” |
Seeing Gemini at 0.7% and Claude at 4.4% on Vectara doesn’t mean “Claude is 6x less accurate than Gemini.” Vectara only tests document-based summarization.
Dataset Difficulty Changes Everything
GPT-4.1 scored 2.0% on the Vectara legacy dataset and 5.6% on the new dataset. When the dataset changes, absolute numbers become incomparable. Only relative rankings within the same dataset are meaningful.
Real-World Hallucination ≠ Benchmark Hallucination
Benchmarks test specific question types in controlled environments. Asking an AI to “analyze the risks in this contract” in your actual work is a completely different scenario from “summarize this news article” in a benchmark. Benchmark numbers reveal directional trends, not precise real-world accuracy rates.
Which Model Should You Choose?
Based on the combined benchmark data, the answer splits by use case.
| Task Type | Recommended Model Family | Rationale |
|---|---|---|
| Document summarization & analysis | Gemini Flash series | #1 in Vectara fidelity; non-reasoning = less over-interpretation |
| Tasks where factual accuracy is critical | Claude series | Lowest hallucination on AA-Omniscience; “refuse rather than guess” behavior |
| Complex reasoning & coding | GPT-5, Claude Opus 4.6 | Reasoning capability is still top-tier (but overkill for summarization) |
| High-volume, speed-sensitive tasks | Lightweight models (Haiku, Flash-Lite, GPT-4.1) | Good cost-performance ratio with acceptable hallucination rates |
No model achieves 0% hallucination. Model selection matters, but so does how you use it: grounding prompts with source material, asking “are you sure?”, cross-verifying critical information. For practical tips on reducing hallucination, see What Is AI Hallucination? Understanding the Mechanism Makes AI Less Scary.
Key Takeaways
The honest answer to “which AI is most accurate” is “it depends on the task.” But this AI hallucination comparison using 2026 benchmark data reveals clear patterns.
First, Gemini leads in document-based tasks. For summarization, analysis, and research, Gemini Flash models consistently show the lowest hallucination rates.
Second, Claude is best at admitting what it doesn’t know. When a wrong answer is worse than no answer, Claude is the safer bet.
Third, reasoning models are smarter but fabricate more. GPT-5, Claude Sonnet 4.6, and Gemini 3 Pro all exceeded 10% hallucination on Vectara. Using a reasoning model for tasks that don’t require reasoning is counterproductive.
Ultimately, using AI well isn’t about finding the single “best model” — it’s about matching the right model to the right task.
References
- Vectara Hallucination Leaderboard (Hugging Face)
- Vectara Blog — Introducing the Next Generation of the Hallucination Leaderboard
- Artificial Analysis — AA-Omniscience Benchmark
- Suprmind — AI Hallucination Rates & Benchmarks 2026
- Suprmind — AI Hallucination Statistics: Research Report 2026
The benchmark figures cited in this article are based on publicly available data as of April 2026. AI models are updated frequently — for the latest numbers, check each benchmark site directly.
답글 남기기