MoE stands for Mixture of Experts. It's a model architecture: certain layers of the network are split into many identically structured sub-networks (the experts), plus a router that decides, for each input, to activate only a few of them.
The tension it addresses is practical — more parameters generally means a stronger model, but every added parameter means more computation on every generation step. MoE lets the total parameter count keep growing while only a small slice is used each time, decoupling "how big the model is" from "how much has to be computed per step."
Grab It in One Sentence First
MoE makes a model "very large but only partly used each time": hundreds of billions of total parameters, a few tens of billions activated per token.
An everyday analogy is triage at a large hospital. The hospital has dozens of departments, but registering as a patient doesn't send you through all of them; triage glances at your symptoms and routes you to the one or two most relevant. The hospital's overall capability is the sum of all departments, while what your visit consumes is just those one or two. The router is the triage desk.
Why This Term Emerged
The idea isn't new — Jacobs et al. proposed mixture of experts back in 1991. It became practical in 2017 when Shazeer et al. applied sparse gating to transformers, after which GShard and Switch Transformer built MoE properly into the transformer's feed-forward layers and made Top-1 and Top-2 routing work at scale.
From 2024 onward, as open-weight models pushed toward trillion-parameter scale, MoE became close to the default choice — because scaling a dense model further raises inference cost linearly, while MoE lets cost grow far more slowly. A widely cited example is DeepSeek-V3: 671 billion total parameters, but only around 37 billion activated per token, meaning under 6% of the model does 100% of the work for that step.
flowchart LR
Token["Input token"] --> Router["Router<br/>scores every expert"]
Router -->|top-k| E1["Expert 1"]
Router -->|top-k| E3["Expert 3"]
Router -.not selected.-> E2["Expert 2"]
Router -.not selected.-> E4["Expert N"]
E1 --> Sum["Weighted sum"]
E3 --> Sum
Sum --> Out["Layer output"]What It Usually Includes
In modern LLMs, an MoE layer generally replaces the feed-forward network inside a transformer block. That layer holds several experts (usually independent FFNs) and a router. The router computes a score per expert for each token, passes them through a softmax to get a probability distribution, and lets only the top-k highest-scoring experts compute; the layer's output is the weighted sum of those k results. In large-scale models, k is often 8.
Several design variants are common. Standard MoE puts every expert into the top-k competition. Shared-expert MoE keeps some experts permanently active and outside routing to carry common knowledge, letting the router choose only among the remaining specialized ones. Fine-grained experts cut experts smaller and activate more of them at once, producing a richer set of combinations at the same parameter count. Expert-choice routing inverts the direction, letting experts pick tokens, which balances load better.
Training brings an unavoidable problem: load balancing. Without constraints, the router gradually concentrates traffic on a handful of experts while the rest go nearly unused — expert collapse. The standard remedies are auxiliary balancing losses and per-expert capacity limits.
The Difference from Dense Models
A traditional transformer's feed-forward layer is dense: every parameter participates in every computation. MoE is sparse: only the selected experts do. The difference shows up in two costs — compute cost is determined by activated parameters, so MoE is cheap; memory cost is determined by total parameters, so MoE is actually hungrier for VRAM, because all experts must stay resident waiting to be called.
That's why MoE isn't unconditionally better. When deployment is memory-constrained, or when implementation simplicity and a smaller footprint matter more, dense models remain a reasonable choice. Coordinating many experts across distributed hardware also adds communication complexity.
Its Relationship to Parameters and Inference Cost
MoE makes "parameter count" a figure that needs qualification. When a model is labeled with hundreds of billions of parameters, ask first whether that's total or activated: the former decides how much VRAM it takes to hold, the latter decides how much computes per token and roughly how fast and how expensive that is. Marketing that gives neither number is of limited use.
The practical effect for users: at a given price point, the model's breadth of knowledge has widened because total parameters went up, while latency and throughput are governed more by activated parameters. That's one reason many models have grown noticeably more capable over the past two years without prices rising much.
Where It's Easy to Misunderstand
The most common misconception is reading "expert" as human-style domain specialization — imagining one expert handles math and another handles Chinese. Routing actually happens at the token level according to patterns that emerge during training; different tokens in one sentence may go to entirely different experts, and the specializations they learn often don't map to any nameable human domain.
The second is "MoE is always cheaper." What it saves is computation per step, not memory or engineering complexity. At small deployment scale, MoE's benefit is often eaten by communication overhead and VRAM pressure.
The third is concluding that lower activation ratios are simply better. The industry has indeed been pushing sparsity down, but research also suggests there's a sweet spot and that going too low costs quality. The optimal activation rate, routing mechanisms better than a simple linear router, and keeping training stable while scaling to hundreds of experts are all still open questions.
How to Decide Whether to Care
If you only call APIs, MoE affects two things: reading spec sheets with total and activated parameters distinguished, and understanding why some models are "huge but cheap." Beyond that it needs no special attention.
If you're self-hosting or running local inference, the trade-offs get very concrete: estimate VRAM from total parameters first, estimate speed from activated parameters second, and confirm your inference framework supports the specific MoE structure you've chosen. When memory is tight, a dense model with a comparable activated-parameter count is often less trouble than forcing an MoE to run.