Context Engineering
Context Engineering
What is context engineering?
- Designing and supplying the 'context' an LLM works from
- Beyond prompts: what information, and how
- Drives RAG and agent quality
Context engineering changes where you look when “the AI can’t answer.” Before, you polished the wording of the question; now you examine the flow of information placed in front of the model: which documents were retrieved, where they were cut off, and what earlier conversation pushed out. Wrong answers usually trace back to the material supplied, not the model, so the practitioner’s job shifts from polishing prompts to designing the information pipeline.
There are usually four layers: system instructions, retrieved documents (RAG), conversation history, and the results of tool or database calls. Recent models accept 120,000 to 200,000 tokens, but information placed in the middle of the window is referenced noticeably less than information at the start or end, so in practice key evidence goes at the very beginning or end. The line with fine-tuning is also clear. If the problem is tone and output format, it is fine-tuning; if the problem is facts and freshness, it is context.
The key to the design is the rules for what to leave out, not just what to put in. For example: split documents into 500–1,000-character chunks and pass only the top 3–5, exclude documents past their revision date from search, and replace conversations longer than 20 turns with a summary. Even with the same question and the same model, accuracy depends on these rules. If you attach the source file name and update date to each answer, then when an answer is wrong you can immediately tell whether retrieval or the model got it wrong.
The most common misconception is 'the window got bigger, so just put everything in.' Paste in a 300-page policy manual and retired clauses end up next to this year's amendments, and the model picks either one to answer with. That is how you get a wrong answer that looks well-sourced. Token costs and response latency also multiply, and as conversations grow, early instructions get diluted by the history. Designing what to leave out matters as much as designing what to put in.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Seolyeong ParkGenerative AI and AX training specialist with 17 years of hands-on experience and 200+ lectures, helping non-technical staff put AI to work right awayA concept that emerged as prompt engineering, RAG, and agents matured, and one that gets more attention as teams put LLMs to work in production.
Consider a hypothetical case. Company A, a building-materials distributor with KRW 180 billion (≈ US$134,670,100) in annual revenue and 240 employees, had been running a Q&A bot for six months to answer questions about internal policies and product price lists. But when the sales team issued quotes based on the bot’s answers, more than ten quotes a month had to be reissued because of wrong prices. Some said, “Let’s switch to a better GPT model,” but the cause was elsewhere.
- Kim, a manager on the IT team, pulled 80 wrong answers and sorted them: nearly half cited a price list that had been retired two years earlier. In the meeting someone said, 'The problem wasn't the model, it was the warehouse.'
- We started by cleaning up the document repository. Of 3,200 files, we removed the 1,100 with no revision date at all, then tagged the rest as 'active/retired' and added update dates.
- We tuned the retrieval settings. Documents are split into 800-character chunks and only the top 5 per question are passed in, with a filter that excludes documents tagged as retired from search entirely.
- Director Lee in Sales said, 'The answer sounds plausible, but I can't tell which document it came from,' so we revised the prompt template to print the source file name and update date at the bottom of each answer.
- Six weeks later, re-measured on the same 100 questions, the error rate fell from 34% to 9%, and average tokens per call dropped from 18,000 to 6,000, cutting monthly API costs to about a third.