# Mixture of experts

> A model built from many sub-networks, where only a few work on each token.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/mixture-of-experts/
- In Spanish: [Mezcla de expertos](https://tidbit.legios.com.ar/es/mixture-of-experts/)

A **mixture of experts** (MoE) is a way to build a neural network so that it can be huge without being slow. Inside some layers of a transformer, instead of one big block that processes every token, there are many smaller blocks called experts, plus a small **router** that decides, for each token, which two or so experts should handle it. The rest sit idle for that token.

That splits the model's size into two numbers. The total parameter count is what the model knows; the active parameter count is what it spends on each token. Mixtral 8x7B, released by Mistral AI in late 2023 as an open-weights model, has roughly 47 billion parameters in total but uses only about 13 billion per token. DeepSeek-V3 has roughly 671 billion in total and about 37 billion active. Many of the largest models today are believed to use some form of MoE, although most labs do not publish their architectures.

The win is cheaper training and faster inference for a given amount of knowledge. The costs are less obvious: all the experts still have to sit in memory, so you need a lot of GPU RAM even if most of it is idle at any moment; the router has to be trained so the load spreads evenly instead of always picking the same favorites; and the experts do not become tidy specialists in medicine or law, as the name suggests. They specialize in patterns of tokens that rarely map to human topics.

## An analogy

A large hospital with dozens of specialists and a triage nurse at the door. Every patient sees only the two doctors they need, so the hospital can employ many experts without every patient paying for all of them. The building still has to have an office for each one.

## Keep in mind

- A router sends each token to a few experts out of many.
- Total parameters can be huge while active parameters per token stay small.
- Cheaper to train and run, but all experts must still fit in memory.
- Experts learn token patterns, not neat human subjects.

## Go deeper

- [Mixture of experts on Wikipedia](https://en.wikipedia.org/wiki/Mixture_of_experts)
- [Mixture of Experts Explained (Hugging Face)](https://huggingface.co/blog/moe)
- [Mixtral of Experts paper (arXiv)](https://arxiv.org/abs/2401.04088)

## Related terms

- [Neural network](https://tidbit.legios.com.ar/neural-network/): A pile of tiny math units that learns patterns from examples.
- [Transformer](https://tidbit.legios.com.ar/transformer/): The neural network design behind modern language models, built around 'attention'.
- [Token](https://tidbit.legios.com.ar/token/): The small chunk of text a language model actually reads.
- [Router](https://tidbit.legios.com.ar/router/): The box that connects your devices to each other and to the internet.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [Mistral AI](https://tidbit.legios.com.ar/mistral-ai/): A French company that builds AI language models, many of them with open weights.
- [Open-weights models](https://tidbit.legios.com.ar/open-weights-models/): AI models whose trained numbers anyone can download and run on their own machines.
- [DeepSeek](https://tidbit.legios.com.ar/deepseek/): A Chinese AI lab whose open-weight models showed that top AI could be built for less.