RAG · Retrieval-Augmented Generation
Retrieval-Augmented Generation
What is RAG?
- Search external knowledge before answering
- Answers grounded in evidence
- Fewer hallucinations, with up-to-date and internal knowledge
The moment you add RAG, the accountability for AI answers shifts. Before, the conclusion was 'the model doesn't know.' Now the cause is 'that document wasn't in the repository' or 'an old version was retrieved.' The factor that drives quality moves from model performance to in-house document management capability. That is why more than half of a rollout project goes not to development but to collecting scattered policies and manuals and sorting out their versions.
It serves a different purpose than fine-tuning and long context. Fine-tuning is strong at changing tone and output format, while the long-context approach of pasting in whole documents suits one-off analysis of a few dozen pages. But when you need to pull just a few pieces of evidence from thousands or tens of thousands of documents, there is no alternative to RAG. Detailed settings also decide the outcome. Documents are typically split into chunks of 300–800 tokens with 10–20% overlap at the edges, and retrieving 3–5 chunks per question is the default. Because semantic search alone misses unique identifiers like 'Approval Form 3-2,' many teams run a hybrid setup that adds keyword search, plus a reranking step.
Once in production, you need separate metrics to monitor it. Build an evaluation set of 30–50 questions that business teams would realistically ask, then check two things separately: whether the needed documents appeared in the search results (retrieval hit rate) and whether the answer matches that evidence. Separating the two metrics shows you where to fix things. If retrieval failed, fix how the documents are split or how titles are organized; if retrieval was right but the answer got muddled, the problem lies in the prompt or the model. Reviewing the log of unanswered questions every week is practically all the maintenance there is.
The most common misconception is that 'if you upload all your documents, it will just answer on its own.' If the 2019 and 2024 editions of the rules of employment sit side by side, the AI will cite a repealed clause, source attached, and employees, seeing the citation, will trust it and act on it. Skipping access design is even riskier. The moment performance reviews or salary tables land in the same repository, a single search lets every employee open them. At a minimum, make it answer 'That is not in the materials' when it can't find evidence, and tag each document with its effective date and access permission so restricted files are filtered out at the search stage.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Ara LimExpert in global leadership, AI, and business communicationIt was first introduced in a 2020 paper by Patrick Lewis and a team at Facebook AI Research (now Meta AI). The core idea is combining a retriever with a generative model.
Let’s look at a hypothetical case. Company A, an auto parts maker with 520 employees, had an HR team of 3 fielding more than 700 internal questions a month. Questions about annual leave, wedding and funeral allowances, and business-trip expenses made up 60% of the total, and searching the policy-manual PDFs to answer them took two to three hours a day. In the first half of 2024, the company decided to build a policy chatbot with RAG.
- The first thing the HR director did was not development but document cleanup. They gathered 180+ policy files scattered across the shared drive, kept only the 42 latest versions, and moved the rest to a discard folder.
- An assistant manager on the information systems team split the 42 documents by clause into a vector database and tagged each with its effective date and access level. Seven files, including the salary table, were isolated so they can be searched only from HR accounts.
- In the first test, the question 'When can I start using annual leave?' returned a repealed 2019 clause. The HR director said, 'Employees just trust it because it cites a source,' and asked to exclude expired documents.
- We built an evaluation set from the 40 questions business teams ask most and ran it every week. After prepending each clause title to its body text and adding keyword search, the retrieval hit rate rose from 61% to 89% in three weeks.
- Three months later, simple HR inquiries fell from 700 to 240 a month, and answers that once took 4 hours on average now come instantly. Because it is set to reply 'Not found in the regulations' when it has no evidence, wrong-answer reports stay at 3 or fewer a month.
- When answers must rely on up-to-date or internal documents
- When data changes often
- When you need to reduce hallucinations
- When you need to change tone or format itself (fine-tuning)
- Pure creative work with no reference documents
- Ultra-low latency where real-time speed is critical