Hallucination
Hallucination
What is a hallucination?
- Plausibly inventing things that aren’t true
- A core risk of generative AI
- Managed through grounding and verification
Understanding hallucination properly changes how you design AI adoption. Instead of asking “What can AI do?”, the question that shapes your workflow becomes who catches errors, and where, when the AI is wrong. Even if drafting gets five times faster, there is no gain if review takes just as long, so in practice the benefit is measured by total time including verification. That is why the safest first task for AI is one with a single correct answer, where mistakes show up immediately.
Hallucinations come in several types. Factual fabrication invents papers, court cases, or statistics that do not exist. Source fabrication gets the content right but makes up the link or page number. Distortion slips sentences into an answer that are not in the evidence document, even when you supplied one. RAG mainly reduces the first two; the third remains. Knowledge cutoffs (what a model cannot know after its training date) and data bias have different causes, so the remedies differ too. Hallucinations become more frequent with longer answers, narrower specialist domains, and more numbers, proper nouns, and dates.
The most common misconception in practice is that “asking for sources makes it safe.” Asking a model that invents sources to cite its sources is not verification. Real verification means checking that the link actually opens and that the document really contains that sentence. The expectation that switching to the latest model solves it is also risky. Errors become less frequent, but the wording gets smoother, so they are harder to spot. In fact, incidents often happen after RAG is added. If you relax review because “it’s evidence-based now,” refund terms that appear nowhere in your customer notice go out, clauses that are not in the contract end up in review summaries, and responsibility falls squarely on the company.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Seolyeong ParkGenerative AI and AX training specialist with 17 years of hands-on experience and 200+ lectures, helping non-technical staff put AI to work right awayThis term came into wide use as generative AI went mainstream and “plausible but wrong answers” emerged as a practical risk. It stems from the fact that a language model does not store correct answers but generates sentences probabilistically.
Consider a hypothetical case. Company A, an industrial materials distributor with 180 employees and annual revenue in the KRW 90 billion (≈ US$67,335,000) range, began using generative AI to draft replies to customer inquiries. Two months in, a client filed a complaint asking Company A to “process this according to the return terms in the email we received,” but no such policy existed at Company A at all.
- When Kim, a manager on the sales support team, reviewed the outgoing emails, three of them contained the sentence “Returns accepted within 14 days of opening,” and no one remembered writing that condition.
- The head of quality assurance checked 120 AI-drafted replies from August one by one against the policy manual and found that 19 of them (15.8%) contained sentences with no internal basis.
- Park, a manager in information systems, uploaded 42 PDFs, including policy manuals, price lists, and standard contract templates, to a searchable repository and fixed an instruction: “If it is not in the provided documents, say only that confirmation from the person in charge is needed.”
- The sales director set a rule that “any sentence containing amounts, deadlines, or legal terms gets a second human review,” and changed the email template so the send button cannot be pressed on such replies until the team lead approves them.
- Three months later, we re-checked 120 replies the same way. Only 2 sentences (1.7%) had no supporting source, and the time to write a reply dropped from 25 minutes to 9 minutes per reply.