# Token

> The small chunk of text a language model actually reads.

- Level: Basic
- Topics: AI
- URL: https://tidbit.legios.com.ar/token/
- In Spanish: [Token](https://tidbit.legios.com.ar/es/token/)

A language model like Claude or ChatGPT does not read letters or whole words. It reads **tokens**: pieces of text that are usually a short word, part of a long word, or a punctuation mark. The word `cat` is one token; `unbelievable` might be three.

Before the model sees your message, a program called a tokenizer splits it into tokens and turns each one into a number. The model works with those numbers, predicts the next token, and the output is turned back into text.

Tokens matter in practice because that is what you pay for and what fits in the model's memory. A page of English is roughly 400 tokens; Spanish and code usually need a bit more.

## An analogy

Think of Lego bricks. You do not build with raw plastic, and you do not buy a whole castle: you work with standard pieces that snap together. Tokens are the bricks of text.

## Keep in mind

- One token is about 3 or 4 English characters on average.
- Prices and limits for AI models are measured in tokens, not words.
- The same text can be more tokens in another language or in code.

## Go deeper

- [Anthropic tokenizer and token counting](https://docs.anthropic.com/en/docs/build-with-claude/token-counting)
- [OpenAI tokenizer playground](https://platform.openai.com/tokenizer)

## Related terms

- [LLM](https://tidbit.legios.com.ar/llm/): A neural network trained on huge amounts of text to predict the next word.
- [Claude](https://tidbit.legios.com.ar/claude/): Anthropic's family of AI models, and the assistant app built on them.
- [ChatGPT](https://tidbit.legios.com.ar/chatgpt/): OpenAI's chat assistant, the app that brought AI chatbots to the general public.
- [Artificial intelligence](https://tidbit.legios.com.ar/artificial-intelligence/): Getting computers to do things that, until recently, only people could do.