LLM Router

An LLM router sits between an application and the models it uses and chooses, for every request, which model should answer. Most applications receive a mix of requests: some are simple, such as classifying a message or answering a common question, and some are hard, such as analyzing a long document or reasoning through a multi-step problem. Sending everything to a frontier model wastes money on easy requests, while sending everything to the cheapest model hurts quality on hard ones. A router makes the choice request by request, a practice often called model routing.

Model routing is one of the most effective ways to reduce AI costs without lowering quality. Because prices between models can differ by a factor of ten or more, moving even a moderate share of requests to a cheaper model can cut spending substantially. For IT financial management, routing turns model choice from a one-time decision into a continuous optimization, and the share of requests handled by each model becomes a key measure of how efficiently an application uses AI.

Definition of LLM Router

An LLM router is software that receives requests intended for a language model and directs each one to a model selected from a pool of available models, based on rules or on a prediction of which model will handle it best at the lowest cost.

Routers are often built into AI gateways, which manage all model traffic for an organization, but they can also be part of an individual application.

Routing Approaches

Rule-based routing uses fixed criteria, such as the type of task, the length of the input or the user's plan, to choose a model. It is simple and predictable but can miss requests that do not fit the rules.

Classifier-based routing uses a small, inexpensive model to assess each request's difficulty or category and choose a model accordingly. It adapts better to varied requests. The classifier itself costs little, since it only needs to judge the request, not answer it.

Cascade routing sends a request first to a cheap model and escalates to a more capable one only if the first answer fails a quality check or the cheap model indicates low confidence. Cascades work well where answers can be checked automatically, such as data that must match a format.

What Routers Optimize

Most routers aim to minimize cost while keeping quality above a set level. Some also consider latency, sending time-sensitive requests to faster models.

Routers can also provide resilience, sending traffic to an alternative model or provider when one is unavailable or rate-limited.

Some routing decisions consider data sensitivity, sending requests that contain confidential information only to approved models or self-hosted deployments.

Importance of LLM Routers in IT Financial Management

Routing directly reduces the average cost per token an application pays, by matching each request to the least expensive model that can handle it.

The savings can be large because request mixes are often dominated by simple requests. If most traffic can be handled by a small model, the cost of the remaining hard requests matters much less.

Routing also makes organizations less dependent on any single model or provider, which helps in negotiations and reduces risk when prices or availability change. Once routing is in place, adopting a new model is a configuration change rather than a rebuild.

Measuring Routing Performance

Key measures include the share of requests sent to each model, the cost per request by model and overall, and quality measures for each route.

Tracking the model mix over time shows whether savings are holding and whether new types of requests are being sent to expensive models.

For cascades, the escalation rate shows how often cheap models fail and how much cost the extra calls add. If escalation rises too high, the cascade may cost more than sending requests straight to the larger model.

Costs and Risks

Routers add some cost and complexity. Classifier-based routing requires an extra model call, although usually a very cheap one.

Poor routing can send hard requests to models that cannot handle them, lowering quality in ways that may not be obvious. Regular evaluation of each route is essential.

Model behavior differs between providers, so prompts and output formats may need adjusting to work consistently across the models a router uses.