Data Labeling · Data Annotation
Data Labeling
What is data labeling?
- Attaching answers (labels) to training data
- The fuel of supervised learning
- Label bias = AI bias
Once data labeling begins, an AI project's center of gravity shifts from the model to the data. Choosing an algorithm takes a day, but the criteria document that defines what counts as a defect, or where 'positive inquiry' begins and ends, takes weeks of back-and-forth with the business side. Labeling is not technical work; it is the work of putting business judgment criteria into writing. If those criteria stay vague while tens of thousands of items are labeled, the workload piles up while performance stays flat. In the end, how much time you can secure from subject-matter experts becomes the real variable in the schedule.
Difficulty and unit cost vary widely by label type. Classification, which assigns one category to a photo, takes a few seconds per image. Bounding boxes, which enclose objects in rectangles, take tens of seconds, and segmentation, which traces outlines pixel by pixel, takes several minutes per image. For text, named-entity annotation marks people and product names, and recently preference labels, which pick the better of two answers, have become more common. Quality is measured with numbers, not gut feeling. The basic method is to run a gold set of several hundred items with confirmed answers past every annotator and measure agreement. Active learning, which picks the data the model finds confusing and labels it first, can sometimes reach the same performance with half the labels or fewer. It is a separate process from preprocessing such as handling missing values or cleaning up formats.
The most common failure in practice is the belief that 'a person labeled it, so it must be right.' Without guidelines for edge cases, different annotators judge the same material differently, and that disagreement hardens into the model's performance ceiling. With a 5% label error rate, 95% accuracy is effectively the cap, yet many teams spend months swapping model architectures without realizing it. When performance stalls, it is faster to inspect the labels before the model. If you order by unit price alone, with no QA conditions, delivery may be fast, but rework costs can exceed the original cost. For data containing personal information, first confirm it has been de-identified before it leaves your organization.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Ara LimExpert in global leadership, AI, and business communicationAn essential preprocessing step for supervised learning, it grew into an industry alongside the building of large datasets such as ImageNet.
Consider a hypothetical case. Company A, an auto-parts manufacturer with 420 employees, had spent eight months on an AI that detects surface scratches. It had trained on 120,000 images, but the detection rate stalled at 78%, and the shop floor started saying, 'People re-check everything anyway, so why use it?'
- When the AI development team lead personally reviewed 300 false positives, they found 41 cases where the same level of fine scratch had been labeled defective in some images and acceptable in others.
- Mr. Park, a manager on the quality assurance team, gathered three inspection foremen and wrote the boundary criteria into a document: 'A scratch under 0.3 mm that is not visible to the naked eye counts as acceptable.' He attached 20 of the most ambiguous photos as reference images.
- We gave eight outsourced labelers 500 images with confirmed answers and measured agreement. It came to 0.62. After two weeks of retraining, only those who scored above 0.87 were assigned to the main work.
- We re-labeled 28,000 of the original 120,000 images where the criteria had been inconsistent. We also reordered the work so the photos the model was least sure about were labeled first, cutting monthly volume from 40,000 images to 12,000.
- After retraining, the detection rate rose from 78% to 93% and the false-positive rate fell from 11% to about 3%. Staff time spent on re-inspection dropped from 6 hours a day to 1.5 hours, and monthly outsourced labeling costs fell by about 40%.