scaling laws for MoE models

a more recently popular LLM model architecture is called Mixture-of-Experts (MoE). unlike dense models, which activate all parameters for every input, MoE models only activate a subset of parameters per token. this design allows models to scale to trillions of parameters efficiently, and it’s been adopted in several of the latest open-source and frontier LLMs.

but how do these bad boys scale?

scaling as we know it

if you have been keeping up with AI innovation and major technical papers that in the past 5 years or so, you would know about “Scaling Laws for Neural Language Models” from OpenAI [1].

the main things to take away from the paper are that a model’s performance, particularly neural language models or dense models, improves as you scale 3 parameters: model size (number of parameters), dataset size (number of training tokens), and compute (training FLOPs). it can be described with this one figure.

all three plots show the relationship between model loss and each of the parameters. each line is basically saying that if you double your resources, you can predict how much your loss will drop.

the key insight is that performance is basically somewhat predictable, and is why many of the major companies are sinking billions into training bigger LLMs.

MoE scaling

so now let’s return to MoE models.

MoE models work differently because only a fraction of the parameters are active per token. because of the decoupling of total model size (by parameter) and compute cost, it boosts efficiency substantially.

as an example, DeepSeekMoE has 16B parameters, and activates 2.8B parameters per token. It still boasts comparable performance to a 7B dense model, which shows a parameter efficiency gain of about 2.5x.

with this efficiency gain due to the decoupling of size and performance, there is a new challenge of predicting the model capacity based on given MoE configurations (expert activation ratio, etc).

AntGroup released “Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models”, which addresses this challenge by introducing a metric called Efficiency Leverage (EL), which quantifies the computational advantage of a given MoE model [2].

it can be calculated as follows.

$$EL(\mathcal{X}_{\text{MoE}} \mid \mathcal{X}_{\text{Dense}}) = \frac{C_{\text{dense}}}{C_{\text{moe}}}$$

the EL of a MoE architecture with respect to a dense baseline is the ratio of their computational costs (C) required for the same performance. as an example, an EL of 2 means that the MoE model only required half the computational cost to get the same performance as the dense baseline.

but what are the actual drivers? AntGroup determined that there are several primary drivers, and some secondary effects. some of the secondary factors include routing quality, expert count, shared expert ratio. the key drivers are:

1. activation ratio (fraction of experts activated per token)
2. granularity (how large each expert is relative to the overall model)
3. compute budget (how much training compute available)

the main scaling law equation is the following.

$$EL(A, G, C) = \hat{A}^{\alpha + \gamma (\log G)^2 + \beta \log G}$$

putting it together, Efficiency Leverage can be expressed as a function of activation ratio (A), granularity (G), and compute (C). the equation demonstrates that as activation ratio decreases, EL increases. the quadratic effect on granularity demonstrates that there is an optimal window for this factor. and like dense models, we see that compute also scales with performance.

similar to the earlier scaling law graph showed, we can represent the relationship in the following graphs.

all three plots show how Efficiency Leverage scales with activation ratio, granularity, and compute budget, illustrating the predictable scaling behaviour of MoE models.

another way to visualize this is with a heatmap, which clearly shows the interaction between activation ratio and granularity. The plot highlights that there is an optimal window for granularity (around 8–12 in this example), where Efficiency Leverage peaks.

closing

evidently, scaling laws tell us why bigger dense models work. but now there is a similar method to demonstrate this for a different kind of architecture. it’s interesting to see that more performance can be unlocked per unit of compute. this suggests that the future of LLM progress may depend as much on architectural innovations like MoE as on simply scaling size and data. but who knows, i still use GPT-5 and Sonnet in my daily flow, still waiting for some Chinese model to really pop off.

---

notes

1. https://arxiv.org/abs/2001.08361
2. https://arxiv.org/abs/2507.17702