Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is mixture of experts (MoE)? Sparse LLMs, explained

Mixture of experts (MoE) routes each token to a few expert networks, so a model can hold 47B parameters but compute with 13B. How routing works, in Python.

What is mixture of experts (MoE)? Sparse LLMs, explained

Mixture of experts (MoE) is a transformer design where each feed-forward layer is split into many parallel “expert” networks, and a small router sends every token to only a few of them. The model stores all the experts but computes each token with a fraction of its weights, which is how Mixtral 8x7B can hold 47B parameters and use only 13B per token.

I ran every snippet below on 2026-10-31 with Python 3.13.11 and no packages.

How does mixture of experts work?

Start with an ordinary transformer block: attention, then a feed-forward network (FFN) that every token passes through. In a dense model that FFN is one big matrix stack, and every token pays for all of it.

An MoE layer replaces that one FFN with N smaller ones plus a router. The Mixtral paper (Jiang et al., January 2024) states it precisely: the output for a token x is y = sum_i Softmax(Top2(x @ W_g))_i * SwiGLU_i(x), summed over the two chosen experts. In words:

  1. The router is a single linear layer, W_g. It turns the token’s hidden vector into one score per expert.
  2. Keep the top k scores (k = 2 in Mixtral) and drop the rest.
  3. Softmax over the survivors to get two weights that sum to 1.
  4. Run the token through those two experts only, and add their outputs using the weights.

Attention, embeddings and layer norms stay shared and dense. Only the FFN is sparse, and the FFN is where most of a transformer’s parameters live, so that’s where sparsity pays.

The idea is older than LLMs. Shazeer et al. introduced the sparsely-gated MoE layer in January 2017, with “up to thousands” of experts between LSTM layers and models up to 137B parameters. Switch Transformers (Fedus, Zoph and Shazeer, 2021) went further and routed each token to a single expert, arguing that k = 1 “preserves model quality, reduces routing computation and performs better”, and trained a 1.6T-parameter model with 2048 experts.

Why do total and active parameters differ?

Because the router decides per token which weights get used. The cleanest way to see it is to count Mixtral’s parameters from the architecture table in its paper: dim 4096, 32 layers, FFN hidden size 14336, 32 query heads, 8 key-value heads, head size 128, vocabulary 32000, 8 experts, top 2.

# moe.py: a top-2 mixture-of-experts layer, and Mixtral 8x7B's parameter count.
# Python 3 standard library only.
import math, random

# 1. Count Mixtral 8x7B's parameters from the config in its paper (Table 1).
dim, layers, hidden, heads, kv_heads, head_dim, vocab = 4096, 32, 14336, 32, 8, 128, 32000
experts, top_k = 8, 2

expert = 3 * dim * hidden                                  # one SwiGLU FFN: w1, w2, w3
attn = dim * heads * head_dim * 2 + dim * kv_heads * head_dim * 2   # q, o, plus k, v (GQA)
router = dim * experts
embed = vocab * dim * 2                                    # input embedding + output head

total = layers * (experts * expert + attn + router) + embed
active = layers * (top_k * expert + attn + router) + embed
print(f"one expert FFN : {expert / 1e6:6.1f}M params per layer")
print(f"total params   : {total / 1e9:6.1f}B  (what you must hold in memory)")
print(f"active params  : {active / 1e9:6.1f}B  (what each token is computed with)")
print(f"active share   : {active / total:6.1%}")
print()

# 2. A toy router: score 8 experts, keep the top 2, softmax over those two.
random.seed(0)
d = 16
W_gate = [[random.gauss(0, 1) for _ in range(d)] for _ in range(experts)]

def route(x):
    logits = [sum(w * v for w, v in zip(row, x)) for row in W_gate]
    top = sorted(range(experts), key=lambda e: logits[e], reverse=True)[:top_k]
    m = max(logits[e] for e in top)
    z = sum(math.exp(logits[e] - m) for e in top)
    return {e: math.exp(logits[e] - m) / z for e in top}

print("token 0 ->", {e: round(g, 3) for e, g in route([random.gauss(0, 1) for _ in range(d)]).items()})

load = [0] * experts
for _ in range(10_000):
    for e in route([random.gauss(0, 1) for _ in range(d)]):
        load[e] += 1
print("tokens per expert over 10,000 tokens (2 slots each):")
print(" ", load, " min/max =", round(min(load) / max(load), 2))
one expert FFN :  176.2M params per layer
total params   :   46.7B  (what you must hold in memory)
active params  :   12.9B  (what each token is computed with)
active share   :  27.6%

token 0 -> {3: 0.875, 6: 0.125}
tokens per expert over 10,000 tokens (2 slots each):
  [2239, 3218, 2434, 2765, 2383, 2564, 2076, 2321]  min/max = 0.65

The arithmetic lands on the paper’s numbers: 46.7B total, 12.9B active (I left out layer norms, which are tiny). It also explains the name. “8x7B” isn’t 56B, because the attention layers and embeddings are shared; only the FFNs are copied eight times.

The second half is the router in miniature. Token 0 goes to experts 3 and 6 with weights 0.875 and 0.125. Over 10,000 random tokens, even an untrained router with random weights sends 3,218 tokens to one expert and 2,076 to another. With real data the skew can get much worse, which is the main engineering problem with MoE.

Do the experts specialize in topics?

Less than the name suggests. Mistral looked for it: in the routing analysis of the Mixtral paper they measured expert choice on different subsets of The Pile and “surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic”. ArXiv papers, PubMed abstracts and PhilPapers looked about the same. What they did find was syntactic: tokens like self in Python and indentation in code tend to hit the same experts, and consecutive tokens often share an expert.

So don’t picture a “math expert” and a “law expert”. Picture a learned hash of token context into buckets, trained so that each bucket gets good at the tokens it receives.

What goes wrong with routing?

Load imbalance. If the router favours a few experts, those experts get more training, which makes the router favour them more. Shazeer’s 2017 paper called this out, and the DeepSeek-V3 report names it “routing collapse”. The classic fix is an auxiliary loss that penalises uneven load. DeepSeek-V3 (December 2024) instead adds a per-expert bias term used only for choosing experts, and nudges it down when an expert is overloaded and up when it’s idle. They call it auxiliary-loss-free load balancing.

Capacity and dropped tokens. In training, each expert gets a fixed batch slot. Tokens that overflow an expert’s capacity get dropped or passed through. Switch Transformers sets each expert’s slot size as tokens per expert times a “capacity factor”; overflow tokens skip the layer through the residual connection, and a bigger factor costs more compute and communication.

Memory, not compute, sets the hardware. The Mixtral paper says it plainly: active parameters are “directly proportional to the inference compute cost”, but serving memory is “proportional to its sparse parameter count, 47B”. It adds that MoE layers are “more suitable for batched workloads”. From a serving seat, that last sentence is the one I’d circle. At batch size 1 on one GPU, you pay 47B of memory and memory bandwidth for 13B of math.

How big do MoE models get now?

Expert counts went up and expert sizes went down. DeepSeek-V3 has 671B total parameters and 37B activated per token. Each MoE layer has 1 shared expert, which every token uses, plus 256 routed experts, of which 8 are picked per token; the first three layers stay dense. The model trained on 14.8T tokens for 2.788M H800 GPU hours, per the report. That’s the base model under DeepSeek R1, the reasoning model I wrote about in January 2025.

Not everyone took the MoE route. Meta’s Llama 3 paper says they chose “a standard dense Transformer model architecture… rather than for a mixture-of-experts model… to maximize training stability”. That’s a fair trade: MoE training is harder to get right, and a dense model is simpler to serve on small hardware.

Mixture of experts vs dense models: which should you run?

It depends on whether you are limited by memory or by compute.

  • Hosted APIs. You don’t see the architecture and don’t need to. My read is that MoE is one reason per-token prices fell, since the provider computes each token with a fraction of the weights.
  • A big GPU server with many users. MoE fits well. Batching keeps all experts busy, and a serving engine like vLLM with expert parallelism spreads experts across GPUs.
  • A laptop or a single small GPU. Count total parameters, not active ones, when you work out how much VRAM you need. A 47B MoE needs the memory of a 47B model. It will generate faster than a dense 47B, since each token reads fewer weights, but it won’t fit where a 13B dense model fits.

The KV cache is a separate memory line item on top of the weights, and MoE does nothing for it. Mixtral uses grouped-query attention (8 KV heads) to keep that part small.

What I would do

When I read a model card, I look for two numbers: total parameters and active parameters. Total tells me what hardware I need. Active tells me roughly what each token costs in compute. If a card only gives one number for an MoE model, I go to the config file and compute the other, the way the script above does.

For self-hosting, I’d pick an MoE model when I have the memory and a steady stream of concurrent requests, and a dense model when I’m memory-bound or serving one user at a time. For fine-tuning, I’d expect more friction with MoE: router behaviour can shift, and not every training tool handles expert layers well. I’d test on the exact framework version before committing.

And I wouldn’t trust “experts” as an explanation of what the model knows. The routing analysis says the split is mostly syntactic, so any story about which expert handles which topic needs evidence from your own traces.

FAQ

What does 8x7B mean in Mixtral 8x7B?

Eight experts per layer, each sized like the FFN of a 7B dense model. Attention and embeddings are shared, so the total is 47B, not 56B, and two experts are active per token, giving 13B active parameters.

Is GPT-4 a mixture of experts model?

OpenAI hasn’t published GPT-4’s architecture, so I can’t say from a primary source. Open-weight MoE models with published configs include Mixtral, DeepSeek-V2 and DeepSeek-V3.

Does MoE reduce memory requirements?

No. All experts must be loaded, because any token can route to any of them. MoE reduces compute per token, which helps speed and serving cost, not the memory needed to hold the weights.

What is the router in mixture of experts?

A small learned linear layer that scores every expert for each token. The top k scores are kept, softmaxed into weights, and the token is sent only to those experts.

Why do MoE models use load balancing?

Without it, the router tends to send most tokens to a few experts, which then train faster and attract even more tokens. Load balancing, through an auxiliary loss or DeepSeek-V3’s bias adjustment, keeps the work spread across experts.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.