Mixture of Experts

Mixture of experts, often abbreviated MoE, is an architecture that lets AI models grow very large without becoming proportionally expensive to run. In a conventional dense model, every parameter takes part in processing every token. In a mixture of experts model, parts of the network are split into many experts, and a small routing component chooses which few experts handle each token. A model might contain hundreds of billions of parameters in total but use only a fraction of them for any given token. Many of today's leading open-weight and proprietary models use this design.

The economic appeal is simple: the model gains the knowledge capacity of a very large network while the compute for each token is closer to that of a much smaller one. That lowers the cost of inference and training relative to a dense model of similar total size. There are trade-offs, however, because all of the experts must still be held in memory, which affects the hardware needed to serve the model. For IT financial management, understanding mixture of experts helps explain model pricing and the real infrastructure costs of hosting these models.

Definition of Mixture of Experts

Mixture of experts is a neural network architecture in which certain layers contain multiple expert sub-networks and a gating or routing function that selects a subset of experts to process each input token. Only the selected experts perform computation for that token.

Models of this kind are often described by two numbers: total parameters, which is the size of the whole model, and active parameters, which is the number used per token. Active parameters drive the compute cost of each token.

How Experts Are Chosen

For each token, the router scores the experts and sends the token to the highest-scoring few, often two or a handful out of dozens or hundreds. The outputs of those experts are then combined.

During training, experts tend to specialize in different patterns, although not in neatly human-recognizable subjects. Training methods encourage balanced use of experts so that no single one becomes a bottleneck.

The routing happens automatically inside the model. Users and applications interact with a mixture of experts model in exactly the same way as with any other language model.

Compute Versus Memory

Because only active parameters are used per token, compute per token is much lower than for a dense model of the same total size. This makes generation faster and cheaper.

However, all experts must be loaded into memory, since any of them may be needed for the next token. Serving the model therefore requires memory for its full size.

This means mixture of experts models can need more GPUs to host than their compute cost suggests, which matters for self-hosting decisions and AI compute planning.

Importance of Mixture of Experts in IT Financial Management

Mixture of experts is one reason AI capability has risen faster than its price. Providers can offer highly capable models at lower per-token prices because each token uses a fraction of the model.

For organizations comparing models, the architecture explains why a very large model can be priced similarly to a smaller dense one, and why its speed may be higher than its size suggests.

For organizations hosting models themselves, the distinction between total and active parameters is essential for estimating hardware needs and cost. Planning from active parameters alone would badly underestimate the memory required.

Hosting Considerations

A mixture of experts model with a large total size needs enough GPU memory to hold all its parameters, plus memory for the KV cache, even though compute needs are modest.

At high traffic, these models can be very efficient, because the large memory investment is spread across many requests that each need little compute.

At low traffic, the memory requirement can make them expensive to host relative to the work done, and an API or a smaller dense model may be more economical.

Pricing and Model Choice

When choosing between models through APIs, mixture of experts models often offer strong quality per dollar, but organizations should compare cost per task rather than assume.

Some mixture of experts models are available as open weights, giving organizations the option to host them, while others are only available through providers.

Quantization is commonly applied to these models to reduce their memory footprint, making self-hosting more practical.