Transformer

The transformer is a way of building neural networks, introduced in 2017 by a team at Google in a paper titled 'Attention Is All You Need'. It is the architecture behind almost every large language model today: the T in GPT stands for it, and Claude, Gemini and Llama are transformers too. It is also used for images, audio, video and even protein structures.

Its central idea is self-attention. Text is first split into tokens, and each token becomes a list of numbers. Then, layer after layer, every token looks at every other token in the context and decides how much each one matters for understanding it. In 'the cat sat because it was tired', attention lets 'it' connect strongly to 'cat'. Many attention 'heads' work in parallel, each picking up different relations: grammar, who did what, which earlier sentence a word refers back to. A language model stacks dozens of these layers and ends by producing probabilities for the next token.

Before transformers, language models mostly read text one word at a time, in order (recurrent networks), which was slow to train and forgot things over long passages. Attention looks at the whole sequence at once, so training can be split across thousands of GPUs running in parallel, and that is what made today's giant models practical. The cost is that comparing every token with every other grows with the square of the text length, one reason context windows are limited and long inputs are expensive.