Prompt caching, batch and model routing: what each actually saves

In brief

Caching cuts repeated input by 90% or more, batch halves work that can wait, and routing saves the price gap. Why the discounts don't add, with a worked bill.

Steve Elliott

Steve Elliott · · 7 min read


On this page
  1. What does prompt caching save, and on what conditions?
  2. When does batch processing pay off?
  3. What do the delayed and premium tiers cost?
  4. What does model routing actually save?
  5. What happens when you combine them?
  6. Which lever should you pull first?
  7. Sources

As of November 2026, prompt caching bills repeated input at 2.5% to 10% of the normal input price, batch processing halves the price of work that can wait up to 24 hours, and model routing saves the price gap between models on the requests a cheaper model can handle. The three don't add up: each discounts what the previous one left, and some providers don't let them stack at all.

In our illustrative bill below, they take $21,000 a month to about $6,400, a 69% cut. Batch's headline 50% on 40% of traffic suggests a further 20% saving once caching is on; it delivers 12%.

What does prompt caching save, and on what conditions?

Prompt caching discounts only input tokens, and writing to the cache often costs more than a normal input token. Terms as of November 2026:

OpenAI (GPT-5.6 and later)Anthropic (Claude API)Google Gemini (Google Cloud and Gemini API)Amazon Bedrock
Cache read price0.1x input on most models; 0.05x on GPT-6.2 Astra and Sol, GPT-6.1 Sol and Terra0.1x on most models; 0.05x on Opus and Sonnet 5.5; 0.025x on Fable and Mythos 5.1 and 5.2Cached input at 10% of the input price (90% off)The model's cache-read rate
Cache write price1.25x for 30 minutes; 2.5x for 12 hours (Responses API)1.25x for 5 minutes; 2x for 1 hourNone listed; explicit caches add storage, about $1 per million tokens per hour on Flash modelsThe model's cache-write rate, which can exceed the input rate
How long it lives30 minutes by default, or 12 hours5 minutes, refreshed free on each hit, or 1 hourExplicit caches default to 1 hour5 minutes, reset on each hit; 1 hour on listed Claude models; 30 minutes on GPT-5.6 and later
Minimum prefix1,024 tokens512 tokens on current models; up to 4,096 on older ones4,096 tokens on Gemini 4 Argon and 3.8 Flash; 2,048 on Gemini 2.5512 to 4,096 tokens, by model
How you turn it onOn by defaultOpt in with a cache control settingImplicit caching on by default for Gemini 2.5 and newerCheckpoints for Claude; implicit for some models

Two conditions decide whether caching pays. First, an exact match: Anthropic says cache hits need "100 percent identical prompt segments," so anything that varies, such as a timestamp, must come after the stable part. Second, reuse inside the cache's lifetime. Anthropic says a 5-minute write breaks even after one cache read and a 1-hour write after two. An AWS post from November 12, 2026 adds the converse: if requests for a prefix arrive more than 5 minutes apart, each one rewrites the cache at 1.25x.

Caching is also best effort: AWS says support doesn't guarantee a hit. We think the cache-read and cache-write counts each response returns are the numbers to watch.

When does batch processing pay off?

In our view, batch inference gives the cleanest discount: 50% at all four providers (on select models at Bedrock), for results within 24 hours rather than seconds. The conditions differ more than the price.

DiscountTurnaroundLimitsStacks with caching?
OpenAI Batch API50%24-hour window only50,000 requests or 200 MB per batchYes: batch cached input is half the standard cached rate
Anthropic Message Batches50%Most batches within 1 hour; expire at 24 hours100,000 requests or 256 MBYes, but hits are best effort: 30% to 98% in Anthropic's figures
Google Gemini on Google Cloud50%24 hours once running; up to 72 hours in queue200,000 requests per jobGemini 3 and later: yes, 95% off cached input. Gemini 2.5: no, the 90% cache discount applies instead
Amazon Bedrock50% for select modelsNo stated typical time; jobs time out at 24 hours by default (up to 168)100,000 records and 5 GB per job (Claude); minimum 100 recordsNo: caching isn't supported with batch inference

OpenAI, Anthropic and Google bill only requests that complete; AWS's documentation doesn't say. Anthropic suggests the 1-hour cache for batches with shared context, because a batch can take longer than 5 minutes. On Bedrock, a prompt with a long shared prefix gets batch or caching, not both.

What do the delayed and premium tiers cost?

OpenAI's background mode offers a 2-hour completion window at 30% below Standard and an 8-hour window at 40% below, in public beta on four GPT-6 models. Standard requests can't reuse cache entries those discounted requests create, so in our view each prompt family should stay on one tier. OpenAI is winding down its Flex tier. Amazon Bedrock's Flex is 50% below Standard but unsupported on Claude Sonnet 5 and Opus 5.

The menu also runs the other way. OpenAI's Fast tier costs twice Standard on GPT-6.2 Sol, and Bedrock's Priority tier carries a 75% premium. In our view, a latency setting chosen once and never revisited can cost more than the discounts save.

What does model routing actually save?

Model routing sends each request to the cheapest model that can answer it well. The saving is the price gap times the share of traffic that moves, minus the cost of what the cheaper model gets wrong. Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens on prompts up to 100,000 tokens (five times that above), against $2 and $10 for Claude Sonnet 5.5; OpenAI prices GPT-6 Luna and GPT-6.2 Sol the same way.

RouteLLM, published at ICLR 2025, reports that its routers "can reduce costs by over 2 times" on public benchmarks without sacrificing response quality; the arXiv version says this holds in certain cases.

Microsoft and AWS sell routers:

  • Microsoft Foundry model router. Balanced mode picks the cheapest model within a quality band of about 1 to 2% of the best; Cost mode widens the band to about 5 to 6%. In most cases Microsoft charges a routing fee per million input tokens on top of the selected model's usage. The effective context window is the smallest in the pool.
  • Amazon Bedrock Intelligent Prompt Routing. It routes between two models in one family, Amazon Nova or Meta Llama. AWS's pricing page says it can cut costs by up to 30% and charges $1 per 1,000 requests. AWS says it is tuned for English prompts only.

A per-request fee can exceed the cheap model's price: Bedrock's $1 per 1,000 requests is $0.001 a request, almost twice the roughly $0.0005 each routed Haiku request costs in the example below. In our view, routing can also undo caching, since a prefix cached for one model doesn't help a request sent to another. Microsoft's session affinity option, in preview, points the same way: it keeps a conversation on one model when that pays off, so cache hits continue.

What happens when you combine them?

Take an illustrative workload on Claude Sonnet 5.5 at Anthropic's list prices as of November 2026: 1 million requests a month, each with a 6,000-token fixed prefix, 2,000 tokens that vary and 500 output tokens. Further illustrative assumptions: a 95% cache hit rate on real-time traffic with 5-minute writes; 40% of requests can wait and move to batch with 1-hour writes and an 80% hit rate; half the real-time traffic is routed to Claude Haiku 5.5, with the same token counts, and 10% of those requests fail a quality check and are rerun on Sonnet. We hold hit rates constant, though splitting traffic across tiers and models usually lowers them.

StepMonthly costChange from previous stepCumulative saving
Baseline: Sonnet 5.5, no levers$21,000
Add prompt caching$10,320-$10,680 (-51%)51%
Move 40% of requests to batch$9,048-$1,272 (-12%)57%
Route half of real-time traffic to Haiku 5.5, rerunning 10%$6,425-$2,623 (-29%)69%

The arithmetic, per million tokens: $2 input, $10 output, $2.50 for 5-minute writes, $4 for 1-hour writes, $0.10 for cache hits.

  1. Baseline. 8,000 million input tokens at $2 ($16,000) plus 500 million output tokens at $10 ($5,000).
  2. Caching. Writes: 300 million tokens at $2.50 ($750). Reads: 5,700 million at $0.10 ($570). Variable input ($4,000) and output ($5,000) don't change.
  3. Batch. Real-time requests cost $6,192. The 400,000 batch requests cost $2,856 at half price, including $960 of 1-hour writes on the 20% that miss, against $4,128 had they stayed real-time. That is a 31% cut, not 50%, because caching had already made most of their input cheap.
  4. Routing. Sonnet's remaining real-time half costs $3,096, Haiku's half $163, and 30,000 reruns on Sonnet $310.

Output, which no cache touches, becomes the largest line: $2,725 of the final $6,425, against $2,180 for uncached input and $1,520 for cache writes and reads. In our view that is the general pattern: once caching works, the next saving comes from shorter answers or cheaper models.

Sonnet 5.5 isn't on Bedrock's batch model list as of November 2026, and caching doesn't work with batch on any Bedrock model. Even where batch is available, at these rates the slice would cost $4,200 in batch, more than the $4,128 it costs cached in real time.

Which lever should you pull first?

We think the order is:

  1. Measure before discounting. Log cached, written and uncached input tokens per request, and track cost per task rather than the rate card.
  2. Fix prompt structure for caching. Stable content first, variable content last. It costs engineering time, not quality.
  3. Move anything that can wait to batch or a delayed tier, after checking whether caching stacks on that provider.
  4. Route last, with evaluations. Routing is the only lever that can change answers, so it needs a quality test, an escalation path and a budget for both.
  5. Recompute when prices change. Google's Gemini 3.8 Flash promotion ends December 31, 2026, and its standard input price then doubles to $1.50 per million tokens.

Steady, high-volume traffic may suit reserved capacity instead. And tokens are only one part of an AI initiative's cost, as our post on tokens vs labor sets out.

Each lever changes who carries a cost and a quality risk, which is why we think the savings belong in the operating model next to the product and the owner that answer for them.

Sources

Frequently asked questions

Do prompt caching and batch discounts stack?

It depends on the provider. OpenAI's batch prices include cached input at half the standard cached rate, and Anthropic says caching and batch discounts can be combined. On Google Cloud, Gemini 3 and later stack to 95% off cached input, but Gemini 2.5 applies only the 90% cache discount. Amazon Bedrock doesn't support prompt caching with batch inference.

How quickly do I need to reuse a prompt for caching to pay?

Within the cache's lifetime, and more than once for long lifetimes. On Anthropic, a 5-minute cache write costs 1.25 times the input price and breaks even after one read; a 1-hour write costs twice the input price and needs two reads. With a 5-minute cache, if requests for the same prefix arrive more than 5 minutes apart, each one rewrites it.

Is the batch API always 50% cheaper?

The list price is 50% off at OpenAI, Anthropic, Google and, for select models, Amazon Bedrock, as of November 2026. The realized saving is often lower. If much of a prompt is already cached, batch halves a smaller amount, and batch cache hit rates are best effort. In our illustrative bill, moving 40% of traffic to batch cut that slice by 31%.

How much can model routing save?

Up to the price gap on the traffic you can move, less the cost of reruns, evaluations and any router fee. On prompts under 100,000 tokens, Claude Haiku 5.5 costs a twentieth as much as Claude Sonnet 5.5, as GPT-6 Luna does against GPT-6.2 Sol. RouteLLM reported cost cuts of over 2 times in certain cases on benchmarks, and AWS claims up to 30% for its router.

Free playbook

Run the enterprise from any AI.

Read The Headless Enterprise
Steve Elliott

Steve Elliott

Published November 22, 2026

More on AI Economics

Tokens, inference cost, cost per task.