Self-Supervised Learning
Self-Supervised Learning
What is self-supervised learning?
- Generating answers from data, without labels
- Learning on its own by filling in blanks
- How LLMs train at scale
What self-supervised learning really changed is not the model architecture but the economics of data. It used to be that using 10,000 records meant labeling the answer by hand 10,000 times, and that labeling budget set the ceiling on project scale. Now human labeling cost shifts to GPU time — internal documents, server logs, and call recordings that sat piled up like warehouse stock become training material as they are, and the bottleneck moves from 'who will do the labeling' to 'how much usable source data do we have'.
There are two main approaches. BERT-style masked prediction hides about 15% of the tokens in a sentence and restores them using context from both sides, which suits embeddings for classification and search. GPT-style next-token prediction sees only what comes before and guesses the next word, which suits generation. In images, contrastive learning, which pairs two versions of the same photo that have been cropped and recolored, uses the same principle. It is easy to confuse with unsupervised learning, but unlike unsupervised methods such as clustering, which have no concept of a correct answer at all, self-supervised learning has a clear correct answer that the data supplies by itself.
Real-world adoption usually follows two stages. First, the model is pre-trained on unlabeled in-house data so it picks up domain terminology and phrasing, and then it is fine-tuned on a labeled dataset of a few thousand examples. Domain pre-training greatly reduces the number of labels needed to reach the same performance, but it reduces them; it does not bring them to zero.
The most common misconception is that 'labels are no longer needed at all.' Teams that believed this and cut their entire labeling budget finished pre-training with no evaluation set to measure performance, and spent months unable to say whether the model had improved or gotten worse. Another is the expectation that feeding in more data will automatically improve results, but duplicate documents and discarded old guidelines are learned just the same. If a price list abolished five years ago is mixed into the document pile, the model will quote that price with great confidence. Humans must stay involved through preprocessing and the evaluation set.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsIt grew out of the need to use large amounts of unlabeled data, and became a core method after the success of large-scale pre-trained models such as BERT and GPT.
Consider a hypothetical case. Company A makes industrial valves, with 940 employees and annual revenue of KRW 410 billion (≈ US$306,748,500). Twenty years of equipment inspection reports and 120,000 failure records existed only as scanned PDFs. When the company set out to build a model that classifies failure types automatically, the outsourced labeling quote came back at KRW 300 million (≈ US$224,500) and six months. The Quality division objected, saying 'for that money, we would rather hire more maintenance staff.'
- Manager Kim on the data analytics team converted all 120,000 records to text, then ran three weeks of pre-training by masking 15% of each sentence and having the model restore it, without attaching a single label.
- The first output was not a classifier but a feel for the terminology. When the model grouped 'packing leak' and 'gland leakage' into the same family, a maintenance foreman with 15 years of experience said, 'This thing actually knows how we talk on the floor.'
- Then two maintenance foremen spent two weeks labeling just 2,000 cases for fine-tuning. What would have cost KRW 300 million (≈ US$224,500) for 120,000 cases came in at KRW 24 million (≈ US$18,000).
- The head of quality set aside 500 evaluation cases and declared that they must never go into training; those 500 became the sole benchmark for every retraining run afterward.
- As a result, failure-type classification accuracy rose from 71% with the old rule-based system to 89%, the average processing time per report fell from 14 minutes to 3, and a rollout planned for six months was cut to seven weeks.