LLMOps
Large Language Model Operations
What is LLMOps?
- A framework for operating LLM services (MLOps for LLMs)
- Built on prompt version control, evaluation sets, and cost monitoring
- Start with an evaluation set, not a tool
What LLMOps really changes is the standard for deployment. Conventional software ships when tests pass, but an LLM service can give different output to the same question each time, so it has to be judged by score, not pass/fail. The release gate therefore shifts from 'build succeeded' to 'evaluation score has not dropped from the previous version,' and what you put under configuration management expands to the combination of prompt, retrieval settings, and model version. The work is managing cases where the code stays the same but service quality changes.
It is more realistic to build evaluation in three layers rather than as one lump. Items a machine can judge, such as JSON format compliance or banned words, get checked exhaustively; ambiguous items such as tone or whether sources are cited are scored by an LLM; and humans sample only the final quality. Each later layer is more accurate but more expensive. The evaluation set doesn’t need to be big at the start: 50–100 real failure cases that triggered complaints are enough to work as regression tests. Because output is non-deterministic, run the same input 3–5 times and look at both the mean and the variance, so you don’t mistake one lucky run for real skill.
Manage cost and latency the same way you manage prompts. Call structure drives spending more than negotiating token prices. Routing each inquiry type to a small or large model and caching repeated questions alone often cut spending by more than half. For latency, the time until the first character appears shapes perceived quality more than total response time, so measure streaming and retrieval-step delays separately to find the bottleneck.
The most common mistake in practice is adopting tools before building an evaluation set. The dashboard shows token usage and latency in neat charts, but nobody can see whether the answers are right, so when quality drops the day a model provider releases a new version, you only find out after customers complain. Trusting an LLM grader blindly is also risky: unless humans periodically verify the grading criteria themselves, answers that sound polished but are actually wrong get high scores. The same goes for the habit of replacing guardrails with a single line at the end of the prompt like 'do not disclose personal information': a system prompt is a request, not a blocking mechanism, so without separate output-side filters and handoff-to-agent rules, incidents go straight to customers.
Sungil KimKAIST and Samsung Electronics alumnus and global pitching expert who coached the winning teams at the UN Tourism and Select USA IR pitch competitions
Ara LimExpert in global leadership, AI, and business communicationA term derived from MLOps (the ML extension of DevOps). After LLM applications exploded following ChatGPT in late 2022, it branched into its own field dealing with challenges unique to LLMs: prompt management, evaluation, and cost optimization.
Consider a hypothetical case. Company A, an office-furniture distributor with 240 employees, had a nine-person CS team handling 6,000 customer inquiries a month, then launched an LLM auto-reply chatbot last October. Answers looked smooth in the demo, but within three months complaints that 'I was given information different from what I ordered' had risen to 12 a week. There was no record of who had changed the prompt or when.
- Manager Kim on the CS team pulled 80 inquiries from the last three months of support logs that had led to complaints over wrong answers, and attached model answers to them. That became Company A’s first evaluation set.
- The dev team moved its scattered prompts into a Git repository and ran an 80-case automatic scoring pass on every change request. They added a rule that blocks any change that scores lower than the previous version.
- Two weeks later, when the model provider released a new version, the accuracy of delivery-tracking answers dropped from 91% to 74%. Automated scoring caught it first and the team rolled back the same day; Kim, the deputy manager, said, 'If customers had noticed first, it would have been a disaster.'
- Park, the operations manager, went through the call logs and found that simple shipping and inventory inquiries made up 62% of the total, so only that segment was routed to a small model, and the monthly API cost fell from KRW 4.2 million (≈ US$3,140) to KRW 1.8 million (≈ US$1,350).
- Refund amount calculations and personal-data verification questions were routed to a human agent instead of getting an automated answer. Every response that got a 👎 was reviewed in a 30-minute session each Friday and added to the evaluation set.
- Three months later the evaluation set had grown to 240 cases. The auto-resolution rate rose from 46% to 68% and wrong-answer complaints fell from 12 a week to 3, so the CS team moved 2 people to returns processing.