Knowledge Distillation
Knowledge Distillation
What is knowledge distillation?
- Large model (teacher) → small model (student)
- Lighter, with the same performance
- Slimmed down alongside quantization
Adopting knowledge distillation changes the questions your team asks. Instead of 'Which model is the smartest?', you ask 'What is the cheapest model that can deliver the capability this task needs?' The large model stops being something you deploy in the service directly and becomes equipment that produces training signals, while the small model handles the actual traffic. That is why a project’s success depends less on picking a model than on designing what to ask the teacher, and how much.
There are two main approaches. White-box distillation, which taps the teacher’s internal probability distribution (logits), only works with open-source models whose weights are accessible. When the internals are off limits, as with commercial APIs, black-box (sequence-level) distillation trains the student on the teacher’s generated answers as if they were the ground truth. Distillation is often confused with quantization and pruning, but those trim the numeric precision or connections of the same model, while distillation trains a new model with a different architecture. The two are not mutually exclusive, and layering quantization on top of a distilled student model is common in real-world deployment.
It doesn’t work on every task, though. The more a task demands general knowledge and long reasoning, the harder it is for the student to keep up. The gap closes quickly on narrowly defined tasks such as inquiry classification, summarization, and internal policy guidance. In practice, teams cut parameters to one-fifth to one-tenth and set the pass line at around 95% of the teacher’s score on the task’s evaluation set. If the student falls short, they first fix how the teacher’s wrong answers are filtered out rather than simply adding more data.
The most common mistake is trusting the teacher without review. If the teacher gives a wrong answer or cites an outdated rule, the student repeats the error with even more confidence, and the cause becomes hard to trace later. Also, if you deploy based only on the overall average score, the model breaks down first on edge cases, which make up 3–5% of inquiries, so you need an evaluation set split by failure type. Some providers prohibit using their commercial API outputs to train your own model, so don’t skip legal review.
Seolyeong ParkGenerative AI and AX training expert with 17 years of hands-on experience and 200+ sessions, helping non-technical staff put AI to work right away
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsGeoffrey Hinton and colleagues established it in a 2015 paper as a method for transferring the knowledge of a large model into a smaller one.
Consider a hypothetical case. Company A is a home appliance maker with KRW 90 billion (≈ US$67,335,000) in annual revenue and 240 employees. Its customer service chatbot runs on an external large-model API, and monthly fees have climbed to KRW 27 million (≈ US$20,200). During peak season, responses slow to an average of 3.2 seconds, and inquiries escalated to human agents pile up. The CS division was told to cut both cost and latency within three months.
- When the head of CS declared, "KRW 27 million (≈ US$20,200) a month is the salary of three agents," the data team lead pulled 180,000 support logs from the past six months and ran them through the teacher model to generate answers along with the supporting policy clauses.
- Two CS managers spent three weeks reviewing a sample of the generated answers and removed 7.4% that were wrong or based on outdated refund policies. The team’s rule was “never pass the teacher’s mistakes on to the student.”
- For the student model, they picked an open-source model with 7 billion parameters and fine-tuned it for 11 days on two in-house GPUs. The infrastructure manager set the pass line at a 1-second response and server costs under KRW 5 million (≈ US$3,740) a month.
- In the first validation round, the 200 most frequently asked questions scored 96% of the teacher’s level, but rare inquiries such as canceling installment payments reached only 71%. So the team distilled 2,300 additional examples for just those types and retrained.
- Eight weeks after the switch, monthly API costs of KRW 27 million (≈ US$20,200) had fallen to KRW 4.2 million (≈ US$3,140) in self-hosted server costs, average response time had dropped from 3.2 to 0.9 seconds, and the agent handoff rate held essentially steady, at 12.8% versus 12.1%.