Quantization · Model Compression
Quantization
What is quantization?
- Lower precision to cut size and compute
- Lighter, performance intact
- The foundation of on-device AI
What quantization really changes is not how smart the model is, but where it can run and what it costs. Loading a 7-billion-parameter model in fp16 takes 14GB of VRAM, so you have to rent a dedicated GPU server. Shrink it to 4 bits and it drops to around 4GB, small enough for an in-house workstation or even a business laptop. The discussion then shifts from 'should we use a better model?' to 'does it fit on our hardware?' Monthly GPU rental fees running to several million KRW are replaced by the one-time cost of a single machine, and this is usually the point where AI adoption begins in air-gapped factories and hospitals that cannot use external APIs.
In practice, there are two main approaches. Post-training quantization (PTQ), which shrinks an already trained model after the fact, finishes in a few hours, so most adoption projects start here. Quantization-aware training (QAT), which builds the precision loss into the training stage, costs more to retrain but preserves quality even at 4-bit and below. In numbers, 8-bit is a safe default with almost no loss, while 4-bit cuts size to a quarter but varies widely by task. You also choose whether to compress only the weights or the activations as well, and the latter is where speed gains actually appear. It is a different axis from pruning or knowledge distillation, so they can be combined.
The most common misconception is 'the benchmark score is unchanged, so quality is unchanged.' Even when the average score drops only 1–2%, errors concentrate in tasks that require exact numbers and proper nouns, such as part numbers, account numbers, and dates, and the gap widens as context gets longer. Hardware is a variable too: on older GPUs that don't support INT4 operations, conversion and restoration steps get in the way and can even make things slower. That is why precision should be chosen not by public metrics but with your own question set built from your company's work data. Skip this validation and you end up with lower costs but a system nobody uses, because the business teams don't trust its answers.
Sungil KimKAIST and Samsung Electronics alumnus and global pitching expert who coached the winning teams at the UN Tourism and Select USA IR pitch competitions
Ara LimExpert in global leadership, AI, and business communicationIt applies the long-established concept of quantization from signal processing and computing to deep learning, and became widespread out of the need to deploy large models efficiently.
Let's look at a hypothetical case: Company A, an auto parts manufacturer with 420 employees and KRW 180 billion (≈ US$134,670,100) in annual revenue. It had been running an internal chatbot that finds equipment manuals and work standards on cloud GPUs for six months, and the monthly cost had climbed to KRW 11.8 million (≈ US$8,830). Worse, Plant 2, which had the most questions, was on a closed network cut off from the outside for security reasons, so it couldn't even use the chatbot.
- When Manager Kim of the Information Strategy Team pulled the cost breakdown and access logs, half of the KRW 11.8 million (≈ US$8,830) monthly bill was for overnight idle hours, and Plant 2 had zero connections.
- An outside consultant uploaded a 4-bit version of the existing 13B model and ran the two side by side for two weeks. VRAM usage dropped from 26GB to 8GB, and it ran as-is on a workstation that had been sitting in the warehouse.
- When the production engineering lead pointed out that the model kept truncating equipment code M-2270 to M-227, Manager Kim collected 200 real questions mixing part numbers and spec values and re-measured accuracy.
- With results split at 81% for 4-bit and 93% for 8-bit, Company A stepped back to 8-bit and improved perceived speed by caching answers to the 30 most frequent questions.
- Six weeks later, the chatbot was running on two workstations inside the plant. Monthly GPU cost fell from KRW 11.8 million (≈ US$8,830) to KRW 1.9 million (≈ US$1,420), and average response time dropped from 3.4 to 2.1 seconds. Plant 2's monthly usage went from 0 to 1,400 requests.