Input and Output Tokens

Every call to a language model has two sides. Input tokens are everything the model reads: the system instructions, the user's question, any conversation history and any documents or data included in the prompt. Output tokens are everything the model writes back, including the visible answer and, for some models, internal reasoning that is billed but not shown. AI providers count the two separately and charge a different price for each.

The distinction matters because input and output tokens behave differently and are reduced in different ways. Output tokens are usually priced several times higher than input tokens, because generating text takes more computation than reading it. Input tokens, however, are often far more numerous, especially in applications that send long documents or conversation histories. For IT financial management, separating the two is essential for explaining AI costs accurately and knowing which optimizations will make the biggest difference.

Definition of Input and Output Tokens

Input tokens, also called prompt tokens, are the AI tokens in the request sent to a model. Output tokens, also called completion tokens, are the tokens in the model's response.

Providers report both counts with each response and publish separate prices for each, usually per million tokens. The total cost of a call is the input tokens multiplied by the input price, plus the output tokens multiplied by the output price.

Why Output Costs More

A model reads all of its input tokens in one parallel pass, which uses GPU capacity efficiently. It then generates output one token at a time, with each new token requiring another pass through the model.

This sequential generation takes more time and occupies more GPU capacity per token, so providers charge more for it. Output prices are commonly three to five times input prices, and sometimes more.

Output tokens also determine how long a user waits for a complete answer, so they affect latency as well as cost. An application that produces long answers will feel slower to users and cost more to run, which gives teams two reasons to keep outputs focused.

Variations in Pricing

Many providers offer a reduced price for cached input, meaning content that was recently sent and can be reused, such as a standard system prompt. Cached input is often billed at a small fraction of the normal input rate.

Some models charge higher rates once a request exceeds a certain length, and some count reasoning tokens as output, even though they do not appear in the answer.

Batch processing, where requests are submitted to run within a longer time window, is often priced at around half the standard rate for both input and output.

Importance of Input and Output Tokens in IT Financial Management

Knowing the split between input and output tokens explains where AI costs come from. Two applications with the same total token count can have very different costs if one is input-heavy and the other output-heavy.

The split also changes which optimizations are worth pursuing. Caching and prompt trimming reduce input costs, while output limits and model choice reduce output costs.

Without this split, teams often optimize the wrong side of the bill. Effort spent trimming prompts may achieve little if most of the cost comes from long answers, and the reverse is equally common.

Input-Heavy and Output-Heavy Workloads

Applications that read a lot and write a little, such as document classification, retrieval-based question answering and data extraction, are input-heavy. Their costs are driven mainly by the size of the documents and context they send.

Applications that write a lot, such as drafting reports, generating code or producing long-form content, are output-heavy. Their costs are driven mainly by output length and the output price.

Agents that call a model repeatedly, each time sending the growing history of the task, tend to become strongly input-heavy as the task goes on.

Forecasting and Pricing

Accurate forecasts need separate assumptions for input and output tokens per request, since each is multiplied by a different price.

Products that pass AI costs on to customers, or that set usage limits, also need to understand the split, because a customer's usage pattern determines their cost per token in practice.

Tracking both measures over time shows whether changes in cost come from longer prompts, longer answers or more requests. Each has a different owner and a different fix, so the distinction speeds up the response when costs rise unexpectedly.