KV Cache

The KV cache is a core part of how language models run efficiently. When a model generates text, each new token depends on all the tokens before it. The model's attention mechanism represents every previous token as a pair of vectors, called keys and values. Rather than recomputing these for the whole sequence every time a new token is produced, the model stores them in the KV cache and reuses them. This makes generation much faster, but it uses a large amount of GPU memory, and that memory grows with every token in the context and every request being served at once.

The KV cache sits at the heart of inference economics. Its size limits how many requests a GPU can serve together and how long the context can be, which in turn determines throughput and cost per token. It is also the mechanism behind prompt caching, where providers reuse stored keys and values across requests to offer discounts. For IT financial management, the KV cache explains why long contexts cost more, why GPU memory matters as much as GPU speed and why some of the most effective inference optimizations focus on managing this memory.

Definition of KV Cache

A KV cache, short for key-value cache, is a store of the attention keys and values a transformer model has computed for previous tokens in a sequence. It is kept in GPU memory during inference and extended by one entry for each new token.

The cache exists only to avoid repeated computation. Without it, generating each token would require reprocessing the entire sequence, making output far slower and more expensive.

Memory Use

The KV cache grows with the length of the context window in use, the number of layers and attention heads in the model, and the number of requests being processed at the same time.

For long contexts, the cache can require more GPU memory than the model's own weights. This limits how many requests fit on a GPU at once and how long each can be.

Memory, rather than raw computation, is often the factor that limits how much work a GPU can do for language model inference, which is why GPUs with more memory command higher prices.

Reuse Across Requests

When many requests share the same opening content, such as a common system prompt, the keys and values for that content are identical. Storing and reusing them avoids recomputing them for every request.

This is the basis of prompt caching, which providers offer as a discounted price for repeated input.

Self-hosted serving software can also reuse cached prefixes across requests, reducing compute and improving response times.

Importance of KV Cache in IT Financial Management

The KV cache links technical design to cost. Because it consumes GPU memory in proportion to context length and concurrent requests, it determines how many users a given amount of hardware can serve.

Long-context applications and agents with growing histories put heavy pressure on the KV cache, reducing throughput and raising cost per request.

Understanding this helps explain pricing that seems puzzling at first, such as higher per-token rates for very long requests or large discounts for cached input. Both reflect what the KV cache costs the provider to hold.

Throughput and Cost per Token

For self-hosted models, more requests served at once means more tokens per second per GPU and a lower cost per token. KV cache memory is often what limits that concurrency.

Techniques that use KV cache memory more efficiently, such as paged memory management, compression of cached values and sharing of common prefixes, allow more requests per GPU.

These improvements directly reduce cost per token, which makes KV cache management a central part of inference optimization.

Context Length Decisions

Every token kept in context occupies KV cache memory for the duration of a request. Applications that send long documents or histories use more memory per request and serve fewer users per GPU.

Reducing context, by retrieving only relevant content or summarizing history, therefore saves not only input tokens but also GPU memory and capacity. For self-hosted services, this can mean needing fewer GPUs for the same number of users.

The same logic explains why providers price long contexts higher: they consume more of the provider's scarcest resource.