Mixture of experts

Why a 501-billion-parameter model uses only 23 billion for each word

A mixture-of-experts model splits part of each layer into many smaller networks called experts. For every token (a word or piece of a word), only a few experts run. The rest stay idle.

Reflection AI's Beam, in today's news, has 501 billion parameters in total (parameters are the numbers the model learned). Only 23 billion of them are active for any one token, so each word costs roughly the compute of a much smaller model.

Mixture of experts

How DeepSeek-V3 picks 8 of its 256 experts for every word

DeepSeek-V3 has 256 routed experts in each mixture-of-experts layer and sends each token to 8 of them. It also has a shared expert that every token uses.

Open models such as Mixtral, Qwen3-235B-A22B and GPT-OSS use the same idea. The 'A22B' in Qwen's name means about 22 billion active parameters out of 235 billion.

Mixture of experts

Each word wakes only the few experts a small router picks

A small learned layer called the router scores every expert for each token, and only the top-k run. Compute per token grows with k, not the total expert count, so a model can hold far more knowledge at about the same cost per word.

The catch: every expert must stay in memory, since the next token might need any of them. Experts also sit on different GPUs, so tokens travel between GPUs and back, which is why AMD's Primus work focuses on that traffic.

If left alone, the router tends to send most tokens to a few favourite experts. Training adds a load-balancing penalty so the work spreads out and every expert learns.