Context Window

The context window is the working memory of a language model. Everything the model takes into account when producing an answer must fit inside it: the system instructions, the user's request, earlier turns of the conversation, documents or data pulled in for reference, and the answer itself as it is generated. Anything outside the window is invisible to the model. Context windows are measured in tokens, and modern models range from tens of thousands of tokens to hundreds of thousands, with some exceeding a million.

Large context windows make new uses possible, such as analyzing a long contract or an entire codebase in one request. They also create a cost risk. Because providers charge for every input token, filling a large context window on every call can make an application expensive very quickly. For IT financial management, the context window is both a capability to use well and one of the main reasons AI costs vary so widely from one application to another.

Definition of Context Window

A context window is the limit on the total number of AI tokens a model can process in a single request, including both input and output. It is a property of each model and is set by its architecture and training.

The window is shared between what goes in and what comes out. A model with a 200,000-token window that receives a 190,000-token prompt has at most 10,000 tokens left for its answer, and many models also set a separate, lower limit on output length.

What Fills the Window

System prompts and instructions are sent with every request. In complex applications they can run to thousands of tokens on their own.

Conversation history accumulates as a chat continues. Unless an application trims or summarizes it, each new message carries all the previous ones with it.

Retrieved content, such as documents, search results or database records, is often the largest part. Agents add the results of tools they have called and the steps they have already taken.

Limits of Long Context

A larger window does not guarantee that a model uses all of the content equally well. Models can miss or underweight information buried in the middle of very long inputs.

Long inputs also take longer to process, which increases the time before a user sees a response.

For these reasons, giving a model less but more relevant context often produces better answers at lower cost than filling the window simply because it is available.

Importance of Context Windows in IT Financial Management

Context use is one of the largest drivers of AI costs. Since every token in the window is billed as input, the amount of context an application sends per request multiplies directly into its monthly bill.

The cost effect grows with usage. An extra 10,000 tokens of context per request looks small in testing but becomes significant across millions of requests.

Some providers also charge higher per-token rates for requests above a certain length, so very long contexts can cost more than their token count alone suggests.

Design Choices and Cost

How an application manages its context is a design decision with direct financial consequences. Sending whole documents, full histories and long instructions is simpler to build but more expensive to run.

Alternatives, such as retrieving only the relevant passages, summarizing history or caching fixed instructions, take more engineering effort but reduce the input and output tokens billed on every call.

Finance and engineering teams benefit from discussing these trade-offs early, before an application's design is fixed and its usage has grown.

Agents and Growing Context

AI agents that work through multi-step tasks often resend their full history with each model call. Context grows with every step, so later calls cost much more than earlier ones.

A task that takes twenty steps can therefore cost many times more than twenty single calls would suggest.

Understanding this pattern is essential for estimating the cost per task of agent-based applications and setting sensible limits on how long agents can run.