Overfitting
Overfitting
What is overfitting?
- Memorizes training data 'too well' and fails on new data
- A student who only crammed the exam questions
- Generalization failure
Once you understand overfitting, what changes is not the model’s performance but the order in which you read the scorecard. You start treating 99% accuracy on training data as an unverified claim rather than an achievement, and the first question in meetings shifts from 'What percentage did it hit?' to 'What data did that number come from?' In other words, the weight of judgment moves from the score itself to the source of the data behind it.
In practice, you split your data into training, validation, and test sets at roughly 6:2:2 and compare training performance with validation performance. A gap of 5 to 10 percentage points or more is the usual threshold for suspecting overfitting. When data is limited, you measure several times while changing how the data is split, as in 5-fold cross-validation, and you reduce the number of variables, apply regularization, or use early stopping, which halts training once the validation score stops improving. However, if both training and validation performance are low, that is underfitting, not overfitting, and the remedy of making the model simpler only makes things worse.
The most common slip in practice is using test data for tuning rather than for measuring performance. If you adjust settings after looking at test scores ten or twenty times, you end up overfitting to that test data yourself, and you are left with a model whose numbers look good and little else. Another is splitting time-series data by random shuffling: records from future time points leak into training, inflating the score, and the model collapses the moment it goes into production. If you approve adoption based on a single line like '96% accuracy', you may end up with a shop floor so worn out by false alarms that it switches the alerts off within six months, so every report should include one line stating what data the number was measured on, and when.
Sungil KimKAIST and Samsung Electronics alumnus and global pitching expert who coached the winning teams at the UN Tourism and Select USA IR pitch competitions
Ara LimExpert in global leadership, AI, and business communicationA foundational concept in statistics and machine learning: a model learns the noise in the data rather than the signal. It is central to how well a model generalizes.
Let's look at a hypothetical case. Company A, an auto-parts manufacturer with 240 employees and annual revenue of KRW 80 (≈ US$1) billion, had a problem: equipment failures on its press line stopped production more than 30 times a year. Management ordered an anomaly-prediction model to be deployed within three months, and the production engineering team and the data analyst worked on it together.
- Manager Kim of the production engineering team built a prediction model from 420,000 sensor-log records covering six months. When it hit 98.4% training accuracy, Kim reported, 'This is good enough to put on the line next week.'
- When Manager Lee, the data analyst, re-ran the model on the most recent two months of logs it had never seen, the hit rate dropped to 61%, and false alarms averaged 14 per day.
- When the team dug into the logs, equipment ID and measurement time had gone straight in as variables, so the model was memorizing a label like 'Unit 3, night shift' rather than any precursor of failure.
- Manager Lee cut the variables from 37 to 12, split the data chronologically into four months for training, one month for validation, and one month for testing, and ran five rounds of validation. As a result, training accuracy fell to 89%.
- In exchange, over two months on the live line the hit rate held at 84% and false alarms fell to 3 per day, and unplanned equipment stoppages dropped from 11 to 4 per month.