Attention mechanism
Attention Mechanism
What is the attention mechanism?
- A technique for focusing on 'what matters right now'
- Captures relationships between distant words
- The heart of Transformers and LLMs
What attention really changed is the problem of 'distance.' Earlier architectures squeezed an entire sentence into a single fixed-length vector and predicted the next word from that alone, so information from the beginning faded the further the text went. Attention lets every token look up every other token directly, treating the 50th word and the 1st word as if they were next to each other. As a result, the bottleneck of the summary vector disappeared, and training could run all at once instead of waiting in sequence.
In practice, it helps to distinguish three variants: self-attention, where words within the same sentence look at each other; cross-attention, which links two separate bodies of text, such as a question and a reference document; and multi-head attention, which splits the same computation into several parallel tracks so each can specialize in grammar, references, or topic. The price is that compute grows with the square of the input length. Double the tokens and attention costs quadruple, so a 32K context and a 128K context come with very different price tags and response times. The longer the window, the steeper the cost curve, and that is where prompt design starts.
The most common mistake in the field is assuming that 'with a big window, you can just put everything in and the model will find what matters.' Attention distributes a limited amount of weight, so the more documents you add, the smaller the share that goes to the sentences that actually matter. It is also repeatedly reported that information in the middle of a long input is reflected less than information at the beginning or end. Reference material is a matter of selection, not volume. One more caution: a high attention weight does not mean that is the model’s basis for its judgment. If you capture an attention map and attach it to an audit report saying 'the model answered by looking at this clause,' you will later have submitted an explanation with no evidence behind it.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsThe self-attention-based Transformer was proposed in the Google researchers’ paper “Attention Is All You Need” (2017) and became the foundation of modern LLMs.
Consider a hypothetical case. Company A, an auto parts maker (1,180 employees, KRW 420 billion (≈ US$314,230,100) in annual revenue), launched an internal Q&A chatbot built on its quality manuals and client spec sheets. Three weeks after launch, the quality assurance team complained that 'we ask the same thing and get a different answer every time.' Finding the cause took six weeks.
- Senior Manager Kim, who led the AI task force, had been pasting the entire 320-page specification into the prompt. The team’s assumption was that if you put it all in, the model will figure out what matters.
- Manager Lee on the quality assurance team personally ran 30 questions. The chatbot answered conditions from the first 20 pages and last 15 pages of the document correctly, but got the heat-treatment tolerance near page 150 wrong in 2 out of 3 tries.
- Park, a senior data engineer brought in as an external advisor, said, 'The model isn’t lazy; its attention is scattered,' and proposed splitting the spec sheets into 40 clause-level chunks.
- They changed the system so that each question requires the applicable process and client name as inputs, and a search step passes in only the top 5 relevant clauses. Average input tokens dropped from 180,000 to 9,000.
- A month later, re-running the same 30 questions showed accuracy up from 61% to 92%, average response time down from 14 seconds to 3 seconds, and monthly API cost down from KRW 7.8 million (≈ US$5,840) to KRW 1.2 million (≈ US$900).