Mixture of experts

A mixture of experts (MoE) is a way to build a neural network so that it can be huge without being slow. Inside some layers of a transformer, instead of one big block that processes every token, there are many smaller blocks called experts, plus a small router that decides, for each token, which two or so experts should handle it. The rest sit idle for that token.

That splits the model's size into two numbers. The total parameter count is what the model knows; the active parameter count is what it spends on each token. Mixtral 8x7B, released by Mistral AI in late 2023 as an open-weights model, has roughly 47 billion parameters in total but uses only about 13 billion per token. DeepSeek-V3 has roughly 671 billion in total and about 37 billion active. Many of the largest models today are believed to use some form of MoE, although most labs do not publish their architectures.

The win is cheaper training and faster inference for a given amount of knowledge. The costs are less obvious: all the experts still have to sit in memory, so you need a lot of GPU RAM even if most of it is idle at any moment; the router has to be trained so the load spreads evenly instead of always picking the same favorites; and the experts do not become tidy specialists in medicine or law, as the name suggests. They specialize in patterns of tokens that rarely map to human topics.