RLHF · Reinforcement Learning from Human Feedback
Reinforcement Learning from Human Feedback
What is RLHF?
- Refining AI with human preferences
- More helpful and safe
- The secret that made ChatGPT usable
A language model that has only finished pre-training simply picks a plausible next word and has no standard for judging which answer is better. What RLHF actually changes is not the model’s knowledge but the source of its judgment criteria. Demands like 'be kind' or 'don’t lie' are hard to score with code, but a person can pick the better of two answers placed side by side in seconds. RLHF turns a problem that is hard to grade into one that is easy to compare, and that is why it could scale.
The standard procedure has three steps. First, supervised fine-tuning (SFT) on thousands of model answers. Second, a reward model trained on tens of thousands of comparisons, where people rank pairs of answers to the same question. Third, reinforcement learning that pushes the model toward higher reward. Recently, lower-cost variants have become common: DPO, which trains directly on preference pairs without a reward model, and RLAIF, which replaces human raters with AI. The boundaries with other techniques are clear too. Prompt engineering leaves the model untouched, and domain fine-tuning adds knowledge and style, but RLHF reorders priorities among what the model already knows, deciding what to bring up first. It does not create knowledge that wasn’t there.
The most common accident in practice is reward hacking. A model doesn’t learn 'good answers'; it learns 'patterns the raters scored highly.' If the panel favors long, polite answers, the model pours out lengthy replies that miss the point, and if it rewards assertive answers, the model confidently makes up things it doesn’t know. Sycophancy, blindly agreeing with whatever the user says, has the same root. If you rush to collect preference data from just 5–6 raters, the tastes of that small group harden into your company standard. Throwing 'pick the better one' at raters with no evaluation guidelines is also risky. If you don’t record what criteria the choices were based on, no one will be able to explain later why the model answers the way it does.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Jaryong KooMarketing PhD and leading authority on AI, data analytics, and personal brandingIt became widely known in 2022 when applied to OpenAI’s InstructGPT and ChatGPT, and has since become the standard alignment technique for matching large language models to human preferences.
Consider a hypothetical case. Company A, a non-life insurer with 740 employees, opened an internal LLM-based support assistant to 120 call center agents. After three months of use, the adoption rate of AI draft replies was only 18%. In team lead meetings, the same complaint kept coming up: 'It sounds polished, but it doesn't point to the policy clauses.'
- Team lead Park of CX Operations recruited 24 agents with 3+ years of experience as a review panel. For each inquiry, two AI-generated answers appeared side by side and the agents picked the better one. It ran 15 minutes a day for 6 weeks.
- Over 6 weeks they collected 4,200 comparison pairs. For the 610 marked 'neither is good enough,' 3 veterans with 15 years of experience rewrote model answers by hand.
- When the team opened the data, agents had chosen answers that lead with the policy clause number about 3 times as often as long, polite ones. Principal Kim summed it up: 'Customers wait for evidence, not apologies.'
- The AI team trained a reward model on this preference data, then tuned the answer model with DPO, and documented evaluation guidelines so that answers without clause citations lose points.
- Team lead Park added a rule to replace half of the 24-member review panel every 8 weeks, so that one agent’s verbal habits would not harden into the company’s standard answer.
- Measured 8 weeks after tuning, answer adoption rose from 18% to 54%, average handling time per case fell from 6 min 40 sec to 5 min 5 sec, and repeat inquiries on the same case dropped 12%.