Explainable AI · XAI
Explainable Artificial Intelligence
What is Explainable AI?
- Explaining why an AI made a decision
- Key to trust, accountability, and compliance
- Opening the black box
Adding XAI changes what an AI system puts out. A screen that used to say only 'Score 712, rejected' now also shows 'the three factors that weighed most in this decision.' What actually changes in practice is not the model but the approval chain and the records. Underwriters compare their own judgment with the model’s reasoning before pressing approve, and customer-facing staff give specific reasons instead of saying 'The system said so.' If explanations aren’t kept in logs, you can’t run post-hoc audits or handle appeals.
Explanation methods fall into two broad groups. One is inherently interpretable models, such as decision trees or linear models, whose structure can be read directly. The other is post-hoc explanation, where tools like SHAP or LIME are added on top of an already trained deep learning model to estimate its reasoning. Within these, a global explanation (“overall, this model weighs the debt ratio most heavily”) and a local explanation (“this is why this customer was rejected”) serve different purposes. Under a framework like the amended Personal Information Protection Act, which gives people the right to request an explanation of, or object to, automated decisions, individuals can ask about their own case, so what you need is the local explanation.
Not every model needs it. In practice, models are sorted by the level of impact their decisions have on people. Where errors can be reversed, as in inventory demand forecasting, teams prioritize performance. Where decisions shape an individual’s opportunities, as in lending, hiring, and insurance underwriting, they choose an explainable structure or keep a local explanation for each case. The quality of the explanation itself must also be verified. Comparing the top factors the model cites with the judgment of subject matter experts often turns up more mismatches than you would expect.
The most common misconception is believing that an explanation is the real reason. A post-hoc explanation is not the model’s internals laid bare but an approximate interpretation, so when the tool or the reference data changes, the order of the top factors shifts too. Tell the same customer “debt ratio” last month and “length of account history” this month, and you have a dispute on the spot. Wording that reads like causation is risky as well. Write “rejected due to a high debt ratio” and the customer hears “approved if I lower it”; if they are rejected again, it comes back as a complaint. An explanation is not proof of fairness; it is the starting point for review.
John KimInstructor John Kim, author of “B2B Marketing Design” and CEO of Nanum Consulting
Junyoung NohConsumer and generational trend analyst & AI content marketing expertAs deep learning’s 'black box' problem came to the fore, XAI research took off in the late 2010s, and its importance is growing alongside debates on AI regulation and accountability.
Consider a hypothetical case. Company A, a consumer finance firm with 240 employees, adopted a credit-loan screening model two years ago and now automatically screens 8,000 applications a month. Approvals got faster, but when rejected customers asked why, the only answer was “the result of a comprehensive credit assessment,” and rejection-related complaints piled up at 120 per quarter. With a regulatory inspection scheduled, the company decided to add an explanation framework over a four-month timeline.
- Director Kim of the Risk Management team read 120 complaints himself and concluded, 'Customers weren’t angry about being rejected; they were angry that they never heard why,' so they decided to record an individual explanation for every rejection.
- The data analytics team used SHAP to extract the top three influencing factors for each case and showed 300 cases to 12 underwriters in a blind test. Only 62% overlapped with the underwriters’ own judgments.
- Digging into the mismatched cases revealed that the telecom-bill delinquency variable was weighing too heavily on one particular age group. The model team removed the variable and retrained, limiting the loss in approval rate to 0.4 percentage points.
- Manager Lee on the legal team objected that “rejected due to a high debt ratio” reads as a promise of approval if the ratio is lowered, so the team standardized the wording to “This factor had the greatest influence on this decision.”
- For the call center, they distributed 18 factor-specific response scripts. They also set a rule that any case where an underwriter disagreed with the model’s explanation goes to the monthly review meeting.
- Four months later, quarterly complaints fell from 120 to 41, the average handling time for rejection-reason inquiries dropped from 9 minutes to just over 3, and the inspection passed the explanation-record retention item with no findings.