Training · Inference
Training / Inference
What is training?
- Training = the stage where the model is taught
- Inference = the stage where a trained model is used
- Different cost profile
Separating training from inference changes the nature of your AI budget. Training is closer to a capital expense: a large one-time cost, then it is done. Inference is closer to an electricity bill, with the meter running every time an employee asks a question. So “How much does this AI cost?” is really two questions. If you don’t calculate the cost of building it and the cost of using it separately, the adoption proposal can sail through approval, only for next year’s operating costs to exceed the budget. Training costs stop once it is finished, but inference costs never stop as long as the service is running. Most companies don’t train models themselves; they only buy inference — and that changes what your evaluation should focus on.
Looking closer, training has layers too. Pre-training, which builds the model itself by feeding it trillions of tokens, is the domain of Big Tech. What a company can realistically do is fine-tuning, which refines tone and output format using a few hundred to a few thousand examples. Frequently changing knowledge, such as internal policy manuals or price lists, is better handled not by training but by finding the relevant document and attaching it at the moment of the question (RAG). Inference is billed per token, and output tokens cost several times more than input tokens, so capping answers at 600 characters versus allowing 2,000 can double your annual bill.
The most common misstep in practice is the expectation that 'if we train it on our data, it will just know everything.' A fine-tuned model picks up the company’s tone, but it doesn’t know last month’s revised price list, and it will still state the old numbers with confidence. If that answer goes straight into a quotation sent to a customer, you have an incident. Another is multiplying pilot costs straight up to a company-wide total. Opening a tool used by ten people to three hundred increases not just the number of users but also queries per person and the length of attached documents, so cost grows far faster than a simple multiple. Plan the rollout on the premise that inference costs keep accruing with every use.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Ara LimExpert in global leadership, AI, and business communicationA basic distinction in machine learning. In the era of large models, the economics of training (a huge one-time cost) versus inference (a cost incurred every time the model is used) has become especially important.
Consider a hypothetical case: Company A, an industrial valve manufacturer with 320 employees and annual revenue of about KRW 85 billion (≈ US$63,594,200). It receives 40 to 60 technical inquiries a day from overseas dealers, and the average time to a first response was 4.2 hours, so complaints were piling up. Management instructed, 'Let’s train the AI on our manuals,' and the Information Strategy team began a six-month review.
- Kim from the Information Strategy team came in with a plan to train a model on all 12,000 pages of English manuals. The outside quote came to KRW 120 million (≈ US$89,800) for three months, and the meeting stopped on the spot.
- The consultant proposed to separate knowledge that changes from tone that doesn’t. Manuals would stay as documents retrieved and attached at query time, and fine-tuning would use only 800 past support replies, just to set the answer format.
- A three-week pilot with 12 agents put monthly inference cost at KRW 180,000, (≈ US$135) and the time to draft a reply fell from 22 minutes to 7 per case. The team lead said, 'At this price, let’s open it company-wide.'
- Manager Lee in Finance recalculated for 140 users and got KRW 2.1 million (≈ US$1,570) a month. Only after capping answers at 600 characters and caching repeated questions, which brought the cost down to about KRW 900,000 (≈ US$675) a month, did Finance approve the expansion.
- Six months later, first-response time had dropped from 4.2 hours to 40 minutes. Fine-tuning was a one-time cost of KRW 3.2 (≈ US$1) million, and monthly inference costs have held steady at KRW 940,000 (≈ US$705). When the price list changes, they simply swap out the document.