Token
The small chunk of text a language model actually reads.
A language model like Claude or ChatGPT does not read letters or whole words. It reads tokens: pieces of text that are usually a short word, part of a long word, or a punctuation mark. The word cat is one token; unbelievable might be three.
Before the model sees your message, a program called a tokenizer splits it into tokens and turns each one into a number. The model works with those numbers, predicts the next token, and the output is turned back into text.
Tokens matter in practice because that is what you pay for and what fits in the model's memory. A page of English is roughly 400 tokens; Spanish and code usually need a bit more.
Keep in mind
- One token is about 3 or 4 English characters on average.
- Prices and limits for AI models are measured in tokens, not words.
- The same text can be more tokens in another language or in code.
