Dike Homme

[태그:] AI hallucination

  • How to Ask AI Better — 5 Rules a Head of Legal Learned the Hard Way

    How to Ask AI Better — 5 Rules a Head of Legal Learned the Hard Way

    Key Takeaway

    AI is not a smart assistant. It’s a wild horse you have to train. How to ask AI better is really how to train it.

    The quality of the answer depends entirely on how you ask. Not the prompt template. Not the role you assign. The question process behind the prompt. I’m a non-lawyer Head of Legal, and I narrowed this down to five rules through hundreds of bad AI outputs in my own review work.

    If you remember nothing else: break tasks down, force the AI to critique itself, demand sources, ask the same question three different ways, and own the final call. The rest of this post is why those five.

    Search “how to ask AI better” and you’ll drown in advice — write better prompts, master harness engineering, role-play your way to better answers. For someone new to AI, that all sounds like astrophysics.

    I’ve been using AI for two years now, starting with ChatGPT, but I only started understanding how it actually behaves a few months back. I treated it like a magic intern that does whatever I say. Then I realized it’s a confident yes-man that fabricates with a straight face. We broke up briefly. Then I came back, because I figured out the real lesson: AI is a wild horse. How you train it decides whether you get a racehorse or an animal that bolts. And I had been demanding racehorse-grade work from a horse I never trained.

    Five rules I beat into shape through hundreds of bad outputs. Not prompt templates — that’s the wrong question. The real question is how to ask AI better, and that lives in the question process, not the wording.

    How to ask AI better — wild horse metaphor
    AI is a wild horse. Train it well and it becomes a racehorse. Skip the training and it bolts wherever it wants.

    Why Prompt Templates Don’t Teach You How to Ask AI Better

    YouTube and Instagram are flooded with “AI 100% productivity” content.

    A lot of Instagram posts run the same hook: “Comment ‘AI’ below and I’ll DM you the secret to making AI smarter.” Like there’s some hidden formula being protected. I’ve actually dropped the keyword and waited for the DM to see what was inside. What you get is a promo for an AI training course bundled with a prompt template — plus the usual generic guidance: “assign a role,” “give clear context,” “include examples.”

    None of it is wrong. But none of them tell you why those rules exist, or when they break. The receiver expects that pasting the template will produce the magic shown in the post, and instead gets the same flat output, just dressed up.

    I made this mistake for months on legal review work. I’d hand AI a contract, assign it the role of “senior attorney,” attach a sample review report, lay out the facts in five-W-one-H format — and the output would still be useless. It would cite case numbers I couldn’t find on any database. It would confidently tell me my company was in the right when I knew we were exposed.

    The problem wasn’t the prompt. It was the question process sitting underneath the prompt.

    Rule 1 — Break Big Asks Into Small Steps

    My most expensive mistake was asking for the deliverable first.

    “Draft a settlement agreement.” “Write me a demand letter.” “How should I respond to this dispute?” Throw a question like that at AI and you get back something abstract, hollow, and worse than what you’d write yourself.

    The fix is to treat AI like a smart but green junior. If you handed a fresh hire a stack of files and said “write me a memo on this dispute,” they’d come back with something pointing in the wrong direction. So you don’t do that. You give them small targets in sequence and let them check in along the way.

    When a dispute lands on my desk, I never ask AI for the report in one shot. I split it into six steps:

    1. Gather the materials — pull all the scattered files into one place, normalize the formats
    2. Summarize each piece — one-line summary per document, sorted by date if relevant
    3. Extract issues — surface the legal issues from those summaries
    4. Research — look up statutes and case law for each issue
    5. Draft — write the report from the research
    6. Self-review — flag any section that’s thin on evidence or relying on inference
    How to ask AI better — Rule 1 step-by-step breakdown vs one-shot ask
    One-shot ask vs. break it down: materials → summary → issues → research → draft → review.

    At each step I add: “If you need to make an assumption, stop and ask me before proceeding.” (You can automate this whole loop using Claude’s agent skills or similar AI agent frameworks, which is what I actually use — but that’s a separate post.)

    When you break the task into smaller targets, AI catches the context, recognizes the patterns inside that context, and gives you depth instead of generic prose. One-shot asks produce mediocre output. Step-by-step asks produce work you can actually use.

    Rule 2 — Make AI Critique Itself

    As I’ve covered in AI Hallucination, Explained, AI doesn’t know the answer and then say it. It produces whatever pattern looks most natural and confident. Sometimes that pattern matches the truth. Sometimes it’s a fabrication wearing the same tone of voice. So any output it hands me has to be assumed wrong until proven otherwise.

    The fastest way to prove it wrong is to make it attack itself.

    Whenever AI gives me a draft, I immediately ask it to find the holes. “Take the position of opposing counsel — what’s the weakest argument here?” “If you were the judge ruling against this brief, what would you flag?” The model splits into a different persona and shreds the work it just produced. AI has no ego, which is exactly why this works.

    I take the revised output and flip the script — argue back from my side. Then have it critique that. After two or three rounds, the durable arguments are the ones still standing, and my actual weak spots are the ones that keep getting hit. Anything that hasn’t been through this self-critique loop, I treat as unfinished.

    Rule 2 — making AI critique its own output
    Split the AI into roles and let them critique each other. The arguments left standing are the durable ones.

    Rule 3 — Always Demand Sources

    The single biggest legal risk with AI is citing case law that doesn’t exist.

    In 2023, two New York attorneys submitted a federal filing containing six AI-fabricated cases. The court imposed a $5,000 sanction on the attorneys and their firm and ruled they had acted in “subjective bad faith” (Mata v. Avianca, Inc., 1:22-cv-01461, S.D.N.Y. 2023). It was the first globally publicized case of AI-fabricated citations reaching the courtroom — and far from the last.

    Stanford’s RegLab measured the underlying rate. In their January 2024 study Large Legal Fictions: Profiling Legal Hallucinations in LLMs, GPT-3.5 hallucinated 69% of randomly queried federal cases. Llama 2 hit 88%. Over 75% of “what was the holding” queries returned false answers. Even commercial legal-AI tools weren’t immune — a follow-up study (May 2024) showed Lexis+ AI at 17% and Westlaw AI at 33% error rates. A more detailed model-by-model breakdown is in my GPT vs Claude vs Gemini hallucination comparison.

    So whenever an answer cites a case or a statute, I run it through the same second prompt:

    “Verify the docket numbers and statutory citations you just used against CourtListener and Justia. Report back any that don’t exist or are misquoted.”

    How to ask AI better — Rule 3 source verification with Stanford RegLab 2024 hallucination rates

    (For Korean readers of this blog: I use law.go.kr and portal.scourt.go.kr for the same check on Korean cases. You can wire either pair into AI directly through MCP — Model Context Protocol — for automated verification, but for most readers, just giving AI the URLs and letting it web-search will catch about half the fakes on the spot.)

    About half the fabricated citations vanish on the second prompt. They get replaced with “could not verify exact case number” — which is the correct output. I verify the survivors manually.

    Anything AI can’t source, I cut. Keep the logic, drop the unverified citations, fill the gaps myself. This single step is the line between AI output that’s usable and AI output that gets you sanctioned.

    The mechanics of why hallucination happens at all are covered in AI Hallucination, Explained.

    Rule 4 — Ask the Same Question Three Different Ways

    Ask the same model the same question twice and the answers shift slightly. Change one word and the conclusion can flip entirely.

    That variability isn’t a bug — it’s a tool. For high-stakes questions I deliberately ask three different angles from the start. For example:

    1. First angle: “Could clause 4 of the attached contract be held unenforceable? On what legal grounds?”
    2. Second angle: “If we adopted clause 4 into our company’s standard contract, what risks does it create?”
    3. Third angle: “If a dispute breaks out around this contract a year from now, what’s the part I’ll most regret?”

    If all three answers point to the same conclusion, the conclusion is solid. If they diverge, the divergence is where the real issue lives — usually a blind spot most people miss.

    Rule 4 — cross-model comparison Claude vs GPT vs Gemini

    This works inside a single model, and it works even better across models — Claude, GPT, Gemini each have different blind spots. I cover model-by-model strengths in How a Head of Legal Built AI Agents Without Writing a Single Line of Code.

    The same rule applies outside legal work — content drafts, investment decisions, even product comparisons. Whatever AI hands you on the first ask is a hypothesis, not a conclusion.

    Rule 5 — You Make the Final Call, Not the AI

    The most important rule, and the one I forget most often.

    You can run five rounds of self-critique, demand sources, ask three different angles, and at the end of it all — the decision is still yours. The moment you outsource that last step, everything upstream collapses.

    What AI is good atWhat you have to do
    Surfacing issuesPicking which issue actually matters
    CritiquingDeciding which critiques to accept
    Compiling sourcesJudging which sources are credible
    Showing varianceCalling the conclusion within the variance
    Producing draftsJudging whether the draft points the right way

    For some teams AI is an automation tool. For a legal team — anywhere data integrity is non-negotiable — AI is one extremely fast teammate. No matter how good your teammate’s draft is, the signature is yours. Forget that, and AI stops being a collaborator and becomes a mistake-generating machine — and the liability lands on you, not it.

    Rule 5 — the human owns the decision, the AI does not
    The accountability for the output sits with you, not the AI. Don’t lose track of that.

    Conclusion — How to Ask AI Better Comes From Knowing Your Work

    These five rules aren’t really AI techniques. They’re work techniques. If you know how to delegate to a junior, you already know how to ask AI well. If you can’t delegate to people, you won’t suddenly delegate well to a machine.

    How to ask AI better tracks how well you understand your own work. If you don’t know your own field’s issues, weak spots, evidence standards, and accountability structure, AI’s output will reflect exactly that gap. If you do know your field deeply, AI multiplies your throughput — in my own work, somewhere between 5x and 10x.

    Your question decides the answer. In the AI era, that sentence carries more weight than ever — and how to ask AI better is the difference between productive collaboration and confident nonsense.

    Further Reading

  • AI Hallucination, Explained — Why AI Makes Things Up and How to Deal With It

    Key Takeaway

    AI doesn’t lie. It was never designed to tell the truth in the first place.

    ChatGPT, Claude, Gemini — these tools are, at their core, an extreme version of your phone’s autocomplete. They predict the next most natural word based on patterns, not facts. Sometimes that prediction is right. Sometimes it produces something that sounds convincing but is completely fabricated. That’s what we call AI hallucination.

    Once you understand this, AI stops being scary. You don’t need to abandon it. You just need to know where it breaks.

    What Is an LLM, Really?

    The technology behind today’s AI chatbots is called an LLM (Large Language Model). The name sounds complex, but what it does is surprisingly simple.

    An LLM reads an enormous amount of text — news articles, research papers, Wikipedia, books, forum posts — and learns one thing: “Given this context, what word is most likely to come next?”

    Think of your phone’s autocomplete. Type “good morning” and it suggests “everyone” or “how are you.” Your phone doesn’t understand greetings. It just knows that “good morning” is statistically followed by certain words.

    An LLM is autocomplete on steroids. Trained on billions of sentences, it predicts the next natural word, appends it, then predicts again — hundreds or thousands of times in a row. That’s how you get those fluent, paragraph-length responses.

    It’s not thinking. It’s pattern-matching at scale.

    Why Does AI Hallucinate? — The Gap Between Pattern and Fact

    Ask an AI “What is the capital of France?” and it answers “Paris.” Correct.

    But did the AI know that Paris is the capital? No. In its training data, “the capital of France is” was overwhelmingly followed by “Paris.” It didn’t look up a fact — it continued a pattern.

    For frequently occurring information, this works remarkably well. The problem emerges when you ask about information that’s rare in the training data: a specific court case number, a niche research paper’s author, a regulation that was recently amended.

    When the pattern is thin, the AI doesn’t stop. It generates something plausible. A case number that looks real but doesn’t exist. An author name that sounds right but belongs to no one. A statute that was never enacted. That’s hallucination — confident fabrication where the AI fills gaps with plausible patterns instead of admitting uncertainty.

    Why Doesn’t AI Just Say “I Don’t Know”?

    This is the question everyone asks, and the answer is surprisingly mechanical.

    According to OpenAI’s 2025 paper “Why Language Models Hallucinate” (arXiv:2509.04664), the root cause lies in how AI models are scored during training:

    • Correct answer → 1 point
    • Wrong answer → 0 points
    • “I don’t know” → 0 points

    Imagine you’re taking a multiple-choice exam. You don’t know the answer to question 17. If you leave it blank, you get 0. If you guess, you have at least a 25% chance of getting it right. Any rational test-taker would guess.

    AI learned the same lesson. During training, guessing always scored equal to or better than abstaining. So the model developed a habit of answering confidently even when it shouldn’t. It’s not trying to deceive — it simply was never given an incentive to say “I’m not sure.”

    Real-World Consequences — This Isn’t Theoretical

    Fake Legal Citations in U.S. Courts

    In 2023, New York attorney Steven Schwartz submitted a legal brief containing six case citations generated by ChatGPT. Every single one was fabricated — fake case names, fake docket numbers, fake rulings. The court sanctioned Schwartz with a $5,000 fine, and the case became the first globally recognized incident of AI hallucination causing real legal consequences.

    It wasn’t an isolated event. In 2025, two attorneys in a Colorado defamation case involving MyPillow CEO Mike Lindell submitted AI-generated briefs containing over 20 fabricated citations, resulting in $3,000 fines each.

    As of 2026, the AI Hallucination Cases Database tracks over 1,300 documented incidents worldwide.

    A Global Pattern

    This is not just an American problem. In South Korea, a judge at Seoul Northern District Court identified fabricated Supreme Court case numbers in a plaintiff’s filing in 2025 and explicitly noted in the ruling that the citations were “AI-generated fakes”. Korean courts detected 10 such cases in 2023, 37 in 2024, and 73 in just the first half of 2025 — prompting the Supreme Court of Korea to establish a task force and add a “fake case number verification” feature to its judicial information portal (per Hankyung reporting).

    The Deeper Risk

    The courtroom cases are dramatic, but the everyday risk is more subtle. AI hallucinations sound exactly like correct answers. The tone doesn’t change when AI is wrong. Without verification, there’s no way to tell a real fact from a fabricated one — and people routinely make decisions based on AI outputs they never checked.

    AI Hallucination Rates Are Dropping Fast — But Context Matters

    The good news: hallucination rates have improved dramatically.

    According to the Vectara Hallucination Leaderboard, the lowest hallucination rate among leading models dropped from 21.8% in 2021 to 0.7% in 2025 — a 96% improvement in four years.

    But these benchmarks measure relatively simple tasks like summarization. In specialized domains, the picture is very different:

    • Medicine: A 2025 MedRxiv study tested AI on 300 clinical cases. Unmitigated hallucination rates reached 64–68%. Even with mitigation prompts, rates remained at 43–45%.
    • Law and Finance: Vectara’s 2026 enterprise benchmark showed hallucination rates above 10% for leading models including GPT-5 and Claude Sonnet 4.5.
    • Person-specific questions: OpenAI’s o3 hallucinated 33% of the time; o4-mini hit 48%.

    (Sources: Suprmind AI Hallucination Benchmarks 2026, All About AI Hallucination Report)

    The takeaway isn’t “don’t use AI.” It’s “know where AI is reliable and where it isn’t.”

    5 Practical Tips to Reduce AI Hallucination

    1. Tell the AI It’s Okay to Say “I Don’t Know”

    As we covered, AI has no built-in incentive to admit uncertainty. But when you explicitly instruct it — “If you’re not sure, say so instead of guessing” — the rate of confident fabrication drops noticeably. One sentence in your prompt can change the behavior.

    2. Ask for Sources

    Requesting citations reduces hallucination by roughly 40%, according to prompt engineering research. When AI is forced to back up its claims, it generates more carefully. And when it does hallucinate a source, you can catch it by checking whether the citation actually exists.

    3. Be Specific

    “Tell me about contract law” invites the AI to fill gaps with patterns. “What are the three requirements for breach of contract damages under Section 390 of the Korean Civil Code?” leaves far less room for fabrication. The narrower your question, the less the model needs to improvise.

    4. Ask AI to Verify Its Own Output

    After getting a response, follow up with: “Check your answer for any inaccuracies or unsupported claims.” AI models can catch their own logical inconsistencies and flag suspicious statements when prompted to self-review.

    For more advanced use, you can build self-verification into your workflow. For example, when using AI for legal research, include an instruction like: “After citing any statute or case, search to confirm it actually exists.” It’s an automated fact-check layer — simple to set up, significant in impact.

    5. Treat AI as a Draft Machine, Not an Answer Machine

    This is the most important mindset shift. AI’s strength is fast first drafts, not factual precision. Every AI output should go through human verification before being acted upon — especially numbers, proper nouns, legal citations, and medical information. If it would be catastrophic to get it wrong, check the original source.

    References