AI Bias
AI Bias
What is AI bias?
- Unfair outcomes caused by skewed data or design
- Biased data makes biased AI
- The risk of automating discrimination
Once you understand AI bias, the question you ask when bringing in a model changes. Instead of "What is the accuracy?" you start by asking "Who does it get wrong?" That is because overall accuracy is the majority group's report card. If 10% of applicants belong to one group, the model can get every one of them wrong and still report 90% overall accuracy. So the core of auditing shifts to breaking performance metrics down by group.
Bias is not a single thing; it enters through different paths. The main ones are historical bias, where past discriminatory decisions harden into ground-truth labels; sampling bias, where a group is underrepresented in the sample to begin with; labeling bias, where evaluators' subjectivity seeps into labels; and feedback-loop bias, where the model's own recommendations return as training data (feedback). Trickier still, there is more than one definition of fairness. Equalizing pass rates across groups and equalizing false-positive rates across groups often cannot both be satisfied mathematically, so which fairness to choose is ultimately a management decision.
Bias should be distinguished from hallucination. Hallucination is a random error in which the model invents facts, whereas bias is directional and reproducible. Enter the same conditions 100 times and it leans the same way 100 times. That is why a reproducible error is caught by looking at the data and the objective function, not the prompt. Conversely, not every statistical difference is unjust discrimination, so you have to judge case by case whether the difference can be explained by job relevance.
The most common misconception in practice is "We removed the gender and age columns, so it's fair." Even when sensitive variables are deleted, proxy variables such as residence, school attended, career gaps, and military service status reconstruct that information. Deleting them can even backfire, leaving you unable to measure bias at all. Another is taking a vendor's "bias testing completed" claim at face value. If you don't ask which country and which group composition the testing was done on, your own applicant pool can produce very different results. The moment it is plugged into an auto-approval line without validation, discrimination is replicated at dozens of cases per second.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsA problem rooted in machine learning’s dependence on data. A series of real-world discrimination cases has made it a core challenge in AI ethics.
Let's look at a hypothetical case. Company A, a retailer with 1,400 employees and annual revenue of KRW 320 (≈ US$1) billion, introduced AI screening for résumé review to handle 12,000 job applications a year. Six months later, the HR team's regular report showed that the résumé pass rate for female applicants had dropped from 31% before adoption to 18%. The discussion began with the question, "The model shows 89% accuracy, so why are we seeing these numbers?"
- When Kim, a manager on the HR planning team, opened the past five years of hiring data used for training, 82% of those hired into logistics and merchandising (MD) roles were men, and the AI had learned that very ratio as a trait of top talent.
- The vendor answered, “Gender is not an input at all,” but when the data analytics lead pulled variable contributions, a career gap of six months or more and graduating from a particular women’s university were acting as penalty factors.
- The CHRO instructed the team, "Don't give me accuracy. Split it by group and report again." When false-positive rates were calculated separately by gender and age band, a 14-point gap between groups appeared.
- The HR team removed career-gap and school variables from training, obtained an additional 4,000 female applicant samples from the past two years, and retrained the model.
- At the same time, we set a rule to use the AI score only as a reference metric and have a recruiter re-review everyone in the bottom 30%, and added a fixed line to the quarterly report showing pass-rate gaps by group.
- Three months later, the pass rate for female applicants recovered to 29% and the gap between groups narrowed from 14 points to 3. One-year retention among new hires actually rose 4 points.