Prompt caching, batch and model routing: what each actually saves
In brief
Caching cuts repeated input by 90% or more, batch halves work that can wait, and routing saves the price gap. Why the discounts don't add, with a worked bill.
Steve Elliott · · 7 min read
On this page
As of November 2026, prompt caching bills repeated input at 2.5% to 10% of the normal input price, batch processing halves the price of work that can wait up to 24 hours, and model routing saves the price gap between models on the requests a cheaper model can handle. The three don't add up: each discounts what the previous one left, and some providers don't let them stack at all.
In our illustrative bill below, they take $21,000 a month to about $6,400, a 69% cut. Batch's headline 50% on 40% of traffic suggests a further 20% saving once caching is on; it delivers 12%.
What does prompt caching save, and on what conditions?
Prompt caching discounts only input tokens, and writing to the cache often costs more than a normal input token. Terms as of November 2026:
| OpenAI (GPT-5.6 and later) | Anthropic (Claude API) | Google Gemini (Google Cloud and Gemini API) | Amazon Bedrock | |
|---|---|---|---|---|
| Cache read price | 0.1x input on most models; 0.05x on GPT-6.2 Astra and Sol, GPT-6.1 Sol and Terra | 0.1x on most models; 0.05x on Opus and Sonnet 5.5; 0.025x on Fable and Mythos 5.1 and 5.2 | Cached input at 10% of the input price (90% off) | The model's cache-read rate |
| Cache write price | 1.25x for 30 minutes; 2.5x for 12 hours (Responses API) | 1.25x for 5 minutes; 2x for 1 hour | None listed; explicit caches add storage, about $1 per million tokens per hour on Flash models | The model's cache-write rate, which can exceed the input rate |
| How long it lives | 30 minutes by default, or 12 hours | 5 minutes, refreshed free on each hit, or 1 hour | Explicit caches default to 1 hour | 5 minutes, reset on each hit; 1 hour on listed Claude models; 30 minutes on GPT-5.6 and later |
| Minimum prefix | 1,024 tokens | 512 tokens on current models; up to 4,096 on older ones | 4,096 tokens on Gemini 4 Argon and 3.8 Flash; 2,048 on Gemini 2.5 | 512 to 4,096 tokens, by model |
| How you turn it on | On by default | Opt in with a cache control setting | Implicit caching on by default for Gemini 2.5 and newer | Checkpoints for Claude; implicit for some models |
Two conditions decide whether caching pays. First, an exact match: Anthropic says cache hits need "100 percent identical prompt segments," so anything that varies, such as a timestamp, must come after the stable part. Second, reuse inside the cache's lifetime. Anthropic says a 5-minute write breaks even after one cache read and a 1-hour write after two. An AWS post from November 12, 2026 adds the converse: if requests for a prefix arrive more than 5 minutes apart, each one rewrites the cache at 1.25x.
Caching is also best effort: AWS says support doesn't guarantee a hit. We think the cache-read and cache-write counts each response returns are the numbers to watch.
When does batch processing pay off?
In our view, batch inference gives the cleanest discount: 50% at all four providers (on select models at Bedrock), for results within 24 hours rather than seconds. The conditions differ more than the price.
| Discount | Turnaround | Limits | Stacks with caching? | |
|---|---|---|---|---|
| OpenAI Batch API | 50% | 24-hour window only | 50,000 requests or 200 MB per batch | Yes: batch cached input is half the standard cached rate |
| Anthropic Message Batches | 50% | Most batches within 1 hour; expire at 24 hours | 100,000 requests or 256 MB | Yes, but hits are best effort: 30% to 98% in Anthropic's figures |
| Google Gemini on Google Cloud | 50% | 24 hours once running; up to 72 hours in queue | 200,000 requests per job | Gemini 3 and later: yes, 95% off cached input. Gemini 2.5: no, the 90% cache discount applies instead |
| Amazon Bedrock | 50% for select models | No stated typical time; jobs time out at 24 hours by default (up to 168) | 100,000 records and 5 GB per job (Claude); minimum 100 records | No: caching isn't supported with batch inference |
OpenAI, Anthropic and Google bill only requests that complete; AWS's documentation doesn't say. Anthropic suggests the 1-hour cache for batches with shared context, because a batch can take longer than 5 minutes. On Bedrock, a prompt with a long shared prefix gets batch or caching, not both.
What do the delayed and premium tiers cost?
OpenAI's background mode offers a 2-hour completion window at 30% below Standard and an 8-hour window at 40% below, in public beta on four GPT-6 models. Standard requests can't reuse cache entries those discounted requests create, so in our view each prompt family should stay on one tier. OpenAI is winding down its Flex tier. Amazon Bedrock's Flex is 50% below Standard but unsupported on Claude Sonnet 5 and Opus 5.
The menu also runs the other way. OpenAI's Fast tier costs twice Standard on GPT-6.2 Sol, and Bedrock's Priority tier carries a 75% premium. In our view, a latency setting chosen once and never revisited can cost more than the discounts save.
What does model routing actually save?
Model routing sends each request to the cheapest model that can answer it well. The saving is the price gap times the share of traffic that moves, minus the cost of what the cheaper model gets wrong. Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens on prompts up to 100,000 tokens (five times that above), against $2 and $10 for Claude Sonnet 5.5; OpenAI prices GPT-6 Luna and GPT-6.2 Sol the same way.
RouteLLM, published at ICLR 2025, reports that its routers "can reduce costs by over 2 times" on public benchmarks without sacrificing response quality; the arXiv version says this holds in certain cases.
Microsoft and AWS sell routers:
- Microsoft Foundry model router. Balanced mode picks the cheapest model within a quality band of about 1 to 2% of the best; Cost mode widens the band to about 5 to 6%. In most cases Microsoft charges a routing fee per million input tokens on top of the selected model's usage. The effective context window is the smallest in the pool.
- Amazon Bedrock Intelligent Prompt Routing. It routes between two models in one family, Amazon Nova or Meta Llama. AWS's pricing page says it can cut costs by up to 30% and charges $1 per 1,000 requests. AWS says it is tuned for English prompts only.
A per-request fee can exceed the cheap model's price: Bedrock's $1 per 1,000 requests is $0.001 a request, almost twice the roughly $0.0005 each routed Haiku request costs in the example below. In our view, routing can also undo caching, since a prefix cached for one model doesn't help a request sent to another. Microsoft's session affinity option, in preview, points the same way: it keeps a conversation on one model when that pays off, so cache hits continue.
What happens when you combine them?
Take an illustrative workload on Claude Sonnet 5.5 at Anthropic's list prices as of November 2026: 1 million requests a month, each with a 6,000-token fixed prefix, 2,000 tokens that vary and 500 output tokens. Further illustrative assumptions: a 95% cache hit rate on real-time traffic with 5-minute writes; 40% of requests can wait and move to batch with 1-hour writes and an 80% hit rate; half the real-time traffic is routed to Claude Haiku 5.5, with the same token counts, and 10% of those requests fail a quality check and are rerun on Sonnet. We hold hit rates constant, though splitting traffic across tiers and models usually lowers them.
| Step | Monthly cost | Change from previous step | Cumulative saving |
|---|---|---|---|
| Baseline: Sonnet 5.5, no levers | $21,000 | ||
| Add prompt caching | $10,320 | -$10,680 (-51%) | 51% |
| Move 40% of requests to batch | $9,048 | -$1,272 (-12%) | 57% |
| Route half of real-time traffic to Haiku 5.5, rerunning 10% | $6,425 | -$2,623 (-29%) | 69% |
The arithmetic, per million tokens: $2 input, $10 output, $2.50 for 5-minute writes, $4 for 1-hour writes, $0.10 for cache hits.
- Baseline. 8,000 million input tokens at $2 ($16,000) plus 500 million output tokens at $10 ($5,000).
- Caching. Writes: 300 million tokens at $2.50 ($750). Reads: 5,700 million at $0.10 ($570). Variable input ($4,000) and output ($5,000) don't change.
- Batch. Real-time requests cost $6,192. The 400,000 batch requests cost $2,856 at half price, including $960 of 1-hour writes on the 20% that miss, against $4,128 had they stayed real-time. That is a 31% cut, not 50%, because caching had already made most of their input cheap.
- Routing. Sonnet's remaining real-time half costs $3,096, Haiku's half $163, and 30,000 reruns on Sonnet $310.
Output, which no cache touches, becomes the largest line: $2,725 of the final $6,425, against $2,180 for uncached input and $1,520 for cache writes and reads. In our view that is the general pattern: once caching works, the next saving comes from shorter answers or cheaper models.
Sonnet 5.5 isn't on Bedrock's batch model list as of November 2026, and caching doesn't work with batch on any Bedrock model. Even where batch is available, at these rates the slice would cost $4,200 in batch, more than the $4,128 it costs cached in real time.
Which lever should you pull first?
We think the order is:
- Measure before discounting. Log cached, written and uncached input tokens per request, and track cost per task rather than the rate card.
- Fix prompt structure for caching. Stable content first, variable content last. It costs engineering time, not quality.
- Move anything that can wait to batch or a delayed tier, after checking whether caching stacks on that provider.
- Route last, with evaluations. Routing is the only lever that can change answers, so it needs a quality test, an escalation path and a budget for both.
- Recompute when prices change. Google's Gemini 3.8 Flash promotion ends December 31, 2026, and its standard input price then doubles to $1.50 per million tokens.
Steady, high-volume traffic may suit reserved capacity instead. And tokens are only one part of an AI initiative's cost, as our post on tokens vs labor sets out.
Each lever changes who carries a cost and a quality risk, which is why we think the savings belong in the operating model next to the product and the owner that answer for them.
Sources
- OpenAI, Prompt caching, accessed November 22, 2026
- OpenAI, Batch API, accessed November 22, 2026
- OpenAI, Background mode, accessed November 22, 2026
- OpenAI, Flex processing, accessed November 22, 2026
- OpenAI, API pricing, accessed November 22, 2026
- OpenAI, GPT-6 Luna model page, accessed November 22, 2026
- Anthropic, Prompt caching, accessed November 22, 2026
- Anthropic, Batch processing, accessed November 22, 2026
- Anthropic, Pricing, accessed November 22, 2026
- Google Cloud, Batch inference with Gemini, updated November 20, 2026
- Google Cloud, Generative AI pricing, accessed November 22, 2026
- Google AI for Developers, Context caching (Interactions API), updated October 23, 2026
- Google AI for Developers, Context caching (generateContent API), updated October 23, 2026
- Google AI for Developers, Gemini Developer API pricing, accessed November 22, 2026
- AWS, Prompt caching for faster model inference, accessed November 22, 2026
- AWS, Amazon Bedrock pricing, accessed November 22, 2026
- AWS, Create a batch inference job, accessed November 22, 2026
- AWS, Supported Regions and models for batch inference, accessed November 22, 2026
- AWS, Amazon Bedrock endpoints and quotas, accessed November 22, 2026
- AWS, Understanding intelligent prompt routing in Amazon Bedrock, accessed November 22, 2026
- AWS Machine Learning Blog, Four Amazon Bedrock inference choices that affect cost, November 12, 2026
- Microsoft Learn, What is model router in Microsoft Foundry?, updated November 18, 2026
- Ong et al., RouteLLM: Learning to Route LLMs from Preference Data, ICLR 2025; arXiv v4, February 23, 2025
Frequently asked questions
Do prompt caching and batch discounts stack?
It depends on the provider. OpenAI's batch prices include cached input at half the standard cached rate, and Anthropic says caching and batch discounts can be combined. On Google Cloud, Gemini 3 and later stack to 95% off cached input, but Gemini 2.5 applies only the 90% cache discount. Amazon Bedrock doesn't support prompt caching with batch inference.
How quickly do I need to reuse a prompt for caching to pay?
Within the cache's lifetime, and more than once for long lifetimes. On Anthropic, a 5-minute cache write costs 1.25 times the input price and breaks even after one read; a 1-hour write costs twice the input price and needs two reads. With a 5-minute cache, if requests for the same prefix arrive more than 5 minutes apart, each one rewrites it.
Is the batch API always 50% cheaper?
The list price is 50% off at OpenAI, Anthropic, Google and, for select models, Amazon Bedrock, as of November 2026. The realized saving is often lower. If much of a prompt is already cached, batch halves a smaller amount, and batch cache hit rates are best effort. In our illustrative bill, moving 40% of traffic to batch cut that slice by 31%.
How much can model routing save?
Up to the price gap on the traffic you can move, less the cost of reruns, evaluations and any router fee. On prompts under 100,000 tokens, Claude Haiku 5.5 costs a twentieth as much as Claude Sonnet 5.5, as GPT-6 Luna does against GPT-6.2 Sol. RouteLLM reported cost cuts of over 2 times in certain cases on benchmarks, and AWS claims up to 30% for its router.
Steve Elliott
Published November 22, 2026
More on AI Economics
Tokens, inference cost, cost per task.
Keep reading
Mastering AI Prompt Generation: A Comprehensive Guide
Learn to craft effective AI prompts. Discover techniques, best practices, and tips to maximize AI output quality and relevance in this comprehensive guide.
Who pays for shared AI? Allocating gateway, evaluation and model costs to teams
Token spend can be metered to teams; reserved capacity, evals, vector databases and the AI team need their own basis. How to split each, with an example.
Self-hosting an open-weight model vs paying per token: where the break-even sits
At Google Cloud prices as of November 2026, self-hosting gpt-oss-120b struggles to beat the same model sold per token. The break-even math, and when it wins.
What Does an IT Manager Do Each Day?
An IT manager runs an organization's technology day to day: keeping systems and networks running, handling support and security, managing the IT team and planning upgrades and budgets.