In the context of AI and large language models, a token refers to a small piece of text that a model uses when processing text. A token might be a character, a word, part of a word, a punctuation mark, or even a stretch of whitespace.
When people read text, they usually feel they're reading sentences and words; before a model processes text, however, it first needs to break the text into tokens and then convert the tokens into numbers. Once you understand tokens, many phenomena become easier to explain: why long documents don't fit, why longer output is slower, and why models are often billed by the token.
Grab It in One Sentence First
A token is the basic unit of text a model uses when reading and generating text.
An everyday analogy is cutting a passage into small building blocks. The model doesn't swallow the whole article at once; it understands and generates step by step, block by block.
Why Models Need Tokens
Computers can't directly "understand a sentence." They first need to turn text into a computable form. Tokenization—the process of turning text into tokens—is exactly the step that cuts text into tokens. After that, each token corresponds to a numeric ID, and what the model actually processes is these sequences of numbers.
flowchart LR
Text["Raw text"] --> Tokenizer["Tokenizer"]
Tokenizer --> Tokens["Token sequence"]
Tokens --> IDs["Numeric IDs"]
IDs --> Model["Model processing"]Different models don't necessarily use the same tokenizer. The same sentence might be 20 tokens in one model and 24 in another. The way Chinese, English, numbers, punctuation, code, and special symbols are split also differs, so a token can't simply be equated with a character count or word count.
Why Tokens Affect Length, Speed, and Cost
The amount of information a model can process at once is usually called the context window, and this window is measured in tokens. The system prompt, the user's question, the conversation history, uploaded materials, retrieved snippets, and the content the model is about to generate all take up tokens.
When content is too long, the system has to make trade-offs: truncate, summarize, chunk, or use RAG to retrieve relevant snippets first. Otherwise the model may not see the earlier information at all, and its answer will naturally lack context.
Tokens also affect cost and speed. Many model APIs count input tokens and output tokens separately. The longer the input, the more the model has to read; the longer the output, the more the model has to generate. Both increase the amount of computation, and may increase the cost.
Where It's Easy to Misunderstand
The most common misconception is treating tokens as a character count. A Chinese sentence may look short but not necessarily have few tokens; a long English word may be split into several subwords; and code and symbols can also consume quite a few tokens.
Another misconception is thinking that the bigger the context window, the more material you should cram in. A large window just means it can hold more information—it doesn't mean the model will automatically grasp the key points. Too much irrelevant information actually interferes with the answer. The truly effective approach is to put the relevant content in and leave the irrelevant content out.
How to Decide Whether to Use It
When handling long documents, don't only think about "dumping it all on the model." A better approach is to filter for relevant content first, and chunk, summarize, or retrieve when necessary. Writing prompts isn't a matter of shorter being better either; it's about being clear, necessary, and not overloaded.
If you use an API, keep an eye on both input and output tokens. Controlling tokens isn't about penny-pinching on word count; it's about letting the model see what it should see while keeping speed, cost, and quality in balance.