# Transformer

> The neural network design behind modern language models, built around 'attention'.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/transformer/
- In Spanish: [Transformer](https://tidbit.legios.com.ar/es/transformer/)

The **transformer** is a way of building neural networks, introduced in 2017 by a team at Google in a paper titled 'Attention Is All You Need'. It is the architecture behind almost every large language model today: the T in GPT stands for it, and Claude, Gemini and Llama are transformers too. It is also used for images, audio, video and even protein structures.

Its central idea is **self-attention**. Text is first split into tokens, and each token becomes a list of numbers. Then, layer after layer, every token looks at every other token in the context and decides how much each one matters for understanding it. In 'the cat sat because it was tired', attention lets 'it' connect strongly to 'cat'. Many attention 'heads' work in parallel, each picking up different relations: grammar, who did what, which earlier sentence a word refers back to. A language model stacks dozens of these layers and ends by producing probabilities for the next token.

Before transformers, language models mostly read text one word at a time, in order (recurrent networks), which was slow to train and forgot things over long passages. Attention looks at the whole sequence at once, so training can be split across thousands of GPUs running in parallel, and that is what made today's giant models practical. The cost is that comparing every token with every other grows with the square of the text length, one reason context windows are limited and long inputs are expensive.

## An analogy

Like a meeting where, before speaking, each person glances at everyone else and decides whose earlier comments matter most to what they are about to say. Everyone does this at the same time, round after round, until the group's understanding is sharp.

## Keep in mind

- Introduced in 2017 in the paper 'Attention Is All You Need'.
- Self-attention lets each token weigh every other token.
- Processes whole sequences in parallel, which made huge models trainable.
- Attention cost grows with the square of the input length.

## Go deeper

- [Transformer (deep learning architecture) on Wikipedia](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture))
- [The Illustrated Transformer (Jay Alammar)](https://jalammar.github.io/illustrated-transformer/)
- [Attention Is All You Need (original paper, arXiv)](https://arxiv.org/abs/1706.03762)

## Related terms

- [Neural network](https://tidbit.legios.com.ar/neural-network/): A pile of tiny math units that learns patterns from examples.
- [Google](https://tidbit.legios.com.ar/google/): The search company that gives most products away and makes its money on ads.
- [LLM](https://tidbit.legios.com.ar/llm/): A neural network trained on huge amounts of text to predict the next word.
- [Claude](https://tidbit.legios.com.ar/claude/): Anthropic's family of AI models, and the assistant app built on them.
- [Gemini](https://tidbit.legios.com.ar/gemini/): Google's family of AI models, and the assistant app that carries the same name.
- [Token](https://tidbit.legios.com.ar/token/): The small chunk of text a language model actually reads.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [GPU](https://tidbit.legios.com.ar/gpu/): The chip that does thousands of simple calculations at once.