Tokenization
Tokenization
What is tokenization?
- Splitting a sentence into tokens
- The first step in how an LLM 'reads' text
- Directly tied to cost and limits
What tokenization really changes is the unit you use to measure text length. People measure documents in characters or pages, but to a model all of that is converted to tokens. Even a single A4 page can have a very different token count from one document to the next; depending on how many tables it has, whether English abbreviations are mixed in, and how line breaks and spaces are handled, the count can differ by nearly double. Whether you are refining a prompt or reviewing a vendor quote, the first change is that the unit of calculation shifts from characters to tokens.
Tokens can be built at the word level, the character level, or with subwords in between, and modern LLMs use subwords almost exclusively. That way, even a new coinage or a typo that isn’t in the dictionary can be split into known pieces and processed. As a working rule of thumb, English averages about 1 token per 4 characters, and Korean about 1–1.5 tokens per character. In other words, the same content in Korean consumes 1.5–2 times as many tokens as in English. It also helps to be clear about the boundary: tokenization stops at splitting text and assigning numbers; giving those numbers meaning and turning them into vectors is the job of embeddings.
The assumption that goes wrong most often in practice is 'characters equal tokens'. It is not unusual for a team to budget a 100,000-character Korean manual as 100,000 tokens, estimate annual costs on that basis, and then receive a bill 1.5 times higher. The context window works the same way. A 128K model does not hold 120,000 Korean characters, and because input and output share the same window, if you push material all the way to the edge of the window, the answer gets cut off midway. Tables, JSON, and Markdown symbols consume far more tokens than their visible length suggests, so if you fix your RAG chunk size by character count, the back half of documents gets silently truncated at the retrieval stage.
John KimInstructor John Kim, author of "B2B Marketing Design" and CEO of Nanum Consulting
Sungil KimGlobal pitching expert from KAIST and Samsung Electronics who coached the winning teams at the UN Tourism and Select USA IR pitch competitionsA basic preprocessing technique in natural language processing (NLP). Subword tokenization (such as BPE) has become the standard approach in modern LLMs.
Consider a hypothetical case. Company A, a home-appliance parts maker with 320 employees, has used an LLM since last October to summarize customer service (CS) records into weekly reports. Within three months, API costs rose to KRW 4.8 million (≈ US$3,590) a month, and for long support cases, the back half of the case was repeatedly missing from the summary.
- When Manager Kim on the information strategy team dug into the call logs, the average input per support case came to 3,800 tokens. The original Korean text was only 1,900 characters, but HTML tags and runs of consecutive spaces had come along with it.
- When Manager Kim counted 200 samples directly with a tokenizer, the numbers confirmed that one Korean character takes 1–2 tokens, and that tags and spaces each take up tokens as well.
- The CS team lead asked, “Wouldn’t it be more accurate to feed in the full original?” But after seeing a tally showing that the passages actually cited in the previous three months of summaries were concentrated in the last 40% of each conversation, the team narrowed the preprocessing scope.
- The team added tag stripping and filters for greetings and auto-reply boilerplate, bringing the average input down to the 1,500-token range. They also changed the pipeline so that any case over 8,000 tokens is split into two calls and merged afterward.
- Two months later, monthly API costs had fallen from KRW 4.8 million (≈ US$3,590) to KRW 1.71 million (≈ US$1,280), and summaries lost to truncated endings dropped from 11 per week to 1.