Tokens per Second

Tokens per second measures how fast an AI model works. For a single user, it describes how quickly an answer appears once the model starts writing: a model producing 50 tokens per second writes a 500-token answer in about ten seconds. For infrastructure, it describes total throughput: how many tokens a GPU, server or cluster can process per second across all the requests it is serving at once. The two views are related but distinct, and both matter for the cost and quality of AI services.

Speed and cost are closely linked in AI. Faster models and faster hardware often cost more per token, while self-hosted infrastructure becomes cheaper per token the more total throughput it achieves. For IT financial management, tokens per second is the measure that connects user experience to infrastructure cost. It shows whether an AI service is fast enough for its users and whether the hardware behind it is producing enough work to justify what it costs.

Definition of Tokens per Second

Tokens per second is a rate measure of AI token processing. It is usually reported separately for output, meaning tokens generated, and for input, meaning tokens read, because models read input far faster than they generate output.

It is commonly paired with time to first token, which measures how long a user waits before any output appears. Together they describe how responsive an AI application feels. Reasoning models can have a long time to first token, because they think before writing, even when their output speed is high.

Per-User Speed

Per-user speed, sometimes called output speed, measures how quickly a model generates tokens for one request. It determines how long users wait for complete answers.

Smaller models and those running on faster or specialized hardware typically generate more tokens per second. Larger models, long contexts and heavy load on a provider's systems slow generation down.

For interactive applications, output speeds well above human reading speed feel instant, while slow generation feels sluggish, especially for long answers.

System Throughput

System throughput measures the total tokens per second processed across all requests on a given piece of infrastructure, such as one GPU server.

Serving many requests together raises total throughput but can lower the speed each individual user sees. Operators balance the two, depending on whether the workload values responsiveness or cost.

Throughput is the key driver of cost per token for self-hosted models, since the hardware costs the same per hour regardless of how many tokens it produces. An idle or poorly tuned server produces few tokens for the same hourly cost.

Importance of Tokens per Second in IT Financial Management

Tokens per second links performance to cost. Faster per-user speed usually costs more, whether through premium pricing tiers, larger hardware or less sharing of capacity, while higher system throughput lowers cost per token.

Organizations need to know how much speed each application really needs, so that they do not pay for speed users will not notice or underinvest where speed affects revenue or productivity.

For self-hosted AI, tokens per second per GPU is one of the most important efficiency measures, alongside GPU utilization. Two deployments on identical hardware can differ several times over in cost per token depending on how well they are tuned.

Speed and Price

Some providers offer faster processing at a higher price, or run the same model on different hardware at different speeds and prices.

Paying for speed makes sense where response time affects outcomes, such as live customer conversations or voice assistants. It makes little sense for background or batch inference work, where nobody is waiting.

Agents that make many sequential model calls are especially sensitive to speed, because delays add up across steps. A task of twenty calls at a few seconds each can take minutes, which affects whether agents are practical for real-time use.

Throughput and Unit Cost

For self-hosted models, cost per token equals the hourly cost of the infrastructure divided by the tokens it processes per hour. Doubling throughput on the same hardware halves the cost per token.

This makes throughput improvements, such as better batching, quantization or more efficient serving software, among the most valuable optimizations for self-hosted AI.

Comparing self-hosted cost per token with API prices requires realistic throughput figures under real traffic, not peak benchmark numbers.