Prompt Caching

Prompt caching reduces the cost of sending the same content to a language model again and again. Many applications start every request with the same material: a long system prompt, a set of examples, a product catalog, a policy document or the earlier part of a conversation. Without caching, the model must reprocess that content in full each time. With prompt caching, the provider keeps the processed version for a short period, and later requests that begin with exactly the same content can reuse it. Cached input is billed at a fraction of the normal input price and is processed faster.

Because input tokens make up most of the volume in many AI applications, prompt caching is one of the largest and simplest savings available. It requires no change to the model and little change to the application beyond structuring prompts so that fixed content comes first. For IT financial management, prompt caching is a standard lever for reducing the cost of AI services, and the share of input served from cache is a useful measure of how efficiently an application uses its model.

Definition of Prompt Caching

Prompt caching is a provider feature that stores the intermediate results of processing a prompt prefix, so that subsequent requests sharing that prefix do not need to recompute it. It relies on the model's KV cache, the internal representation built as the model reads each token.

Caching applies to the beginning of a prompt. A request benefits only if its opening content matches a cached prefix exactly, which is why fixed content should be placed before content that changes. Even a single different token near the start of a prompt means the rest of it cannot be read from cache.

How Providers Charge

Providers bill cache reads, meaning tokens served from the cache, at a large discount to normal input prices, often between half and a tenth of the standard rate depending on the provider.

Some providers charge a small premium to write content into the cache the first time, while others cache automatically at no extra charge.

Cached content expires after a short period without use, commonly a few minutes, although some providers offer longer cache lifetimes at a higher write cost.

What to Cache

Good candidates include long system prompts and instructions, examples used to guide the model, tool definitions and reference documents that many requests share.

In conversations, earlier turns can be cached so that each new message reuses the history rather than reprocessing it.

For agents, the fixed instructions and the stable early part of the task context can be cached across steps. This matters because agents resend their context with every step, so caching it can reduce the cost of long tasks considerably.

Importance of Prompt Caching in IT Financial Management

Prompt caching can cut input costs dramatically for applications with large, repeated prompts. Since input tokens often make up most of the tokens billed, the effect on total cost can be substantial.

It also reduces latency, because cached content does not need to be processed again. This improves user experience at the same time as reducing cost, which makes it easy to justify.

Despite this, many applications are built without caching in mind, sending variable content first or making small changes to instructions with every request, which prevents cache hits. Reviewing prompt structure is therefore one of the first steps in any AI cost review.

Measuring Cache Performance

The cache hit rate, meaning the share of input tokens served from cache, shows how well an application uses caching. Providers report cached and uncached tokens separately in their usage data.

Tracking the hit rate alongside cost per token shows how caching affects the effective price an organization pays.

Falling hit rates can signal changes in prompt design, such as dynamic content added to the start of the prompt, that are quietly increasing costs.

Limits of Prompt Caching

Caching helps only when the same prefix is reused within the cache lifetime. Applications with low traffic, or with prompts that differ from the very first token, gain little.

Cache write premiums can make caching slightly more expensive for content that is rarely reused, so it should be applied deliberately.

Prompt caching is different from semantic caching, which stores and reuses whole answers to similar questions. The two can be used together.