Who pays for shared AI? Allocating gateway, evaluation and model costs to teams
In brief
Token spend can be metered to teams; reserved capacity, evals, vector databases and the AI team need their own basis. How to split each, with an example.
Steve Elliott · · 7 min read
On this page
- Why does a central AI gateway make attribution harder?
- How do you attribute token spend to teams?
- Which shared AI costs sit outside per-request token metering?
- Which allocation method fits each shared AI cost?
- What does a worked allocation look like?
- Should you show teams these costs or charge them?
- Sources
We think teams that consume a shared AI service should pay for their own tokens by direct metering, and for the shared costs around those tokens by a stated rule per cost type. Pay-as-you-go token spend maps straight onto the invoice; the gateway, eval runs, vector databases, reserved capacity and the central AI team each need a stated basis instead of one blanket percentage.
A common shortcut, spreading everything in proportion to token spend, is easy to run and, in our view, misleading. It charges teams for capacity they never touch and for capabilities no team uses yet. Below: what provider attribution features record, a method per cost type, and an illustrative allocation.
Why does a central AI gateway make attribution harder?
Routing every model call through one AI gateway helps policy and hurts native attribution unless you design for it. AWS states the problem directly: when a gateway calls Amazon Bedrock on behalf of many callers, Bedrock records the gateway's IAM role as the caller's identity. Every team's spend then shows up as the gateway's.
The gateway is also where the fix goes. AWS's cost attribution guidance says a shared client wrapper or gateway is the most reliable place to stamp request metadata, because Bedrock does not reject calls that omit per-call tags. The FinOps Foundation's paper on direct model provider APIs, updated October 28, 2026, makes the same case: a proxy or gateway attaches the organization's own metadata, such as application, feature and cost center, to every call. AWS adds that you can use service control policies to deny Bedrock access from outside the gateway's role, so the gateway becomes the only call path.
How do you attribute token spend to teams?
The FinOps Foundation's paper sets out three steps: govern API keys, use provider-native attribution, then add a proxy for richer metadata. Its FinOps for AI overview, also updated October 28, 2026, says to map each API key to one team, application and cost center. What each provider records, as of November 2026:
- OpenAI. The Costs endpoint groups spend by project ID, user ID, line item, API key ID and API source, in daily buckets only.
- Anthropic. The Usage and Cost Analytics API, in beta, breaks cost down by workspace, API key (for Console organizations), model, service tier and the identity behind each request, in minute, hourly or daily buckets. One gap matters for allocation: requests served under a Priority Tier commitment show zero token cost, and the commitment fee is billed separately and isn't in this data.
- Amazon Bedrock. IAM principal attribution, application inference profiles, projects and workspaces deliver billed dollars to Cost Explorer and CUR 2.0, at a finest grain of usage type per day. Per-request metadata, up to 16 entries per call, lands only in invocation logs as token counts that you convert to cost. Cost allocation tags must be activated, aren't retroactive, and can take up to 24 hours to appear.
- Microsoft Foundry. Tags on model charges assign them to a project, a model deployment and, for agent requests, an agent. Microsoft notes that breakdowns include only tagged charges, so they usually add up to less than the total.
- Google Cloud. Request labels (in the body for Gemini, a header for partner models) are forwarded to Cloud Billing. They don't reach billing on Provisioned Throughput traffic. Each label key can hold at most 1,000 unique values over the life of the billing account, and Google may drop a key that exceeds it.
Gateways have limits too: Azure API Management's token metric policy allows up to five custom dimensions, each limited to 100 unique values; beyond that, or past 1,000 time series, new values are silently dropped.
In our view, two rules follow. Keep allocation keys low in cardinality (team, application, environment, cost center) and put session or experiment IDs in logs; AWS gives the same advice. And reconcile log-derived cost against CUR at the model and usage-type level, which AWS recommends doing periodically and we would do monthly, because log-derived cost misses discounts, commitments, batch pricing and provisioned throughput unless you model them.
Which shared AI costs sit outside per-request token metering?
Infrastructure around the model, and the people who run it. The FinOps Foundation paper says the surrounding infrastructure "often represents 40 to 60 percent of total feature spend" in production retrieval and agent systems. It lists vector databases, embedding generation, rerankers, orchestration, caches, cross-region data transfer and observability. The paper cites no source for the range; we treat it as practitioner experience.
It recommends two attribution streams, token spend from the proxy and the rest from cloud and SaaS billing, joined on a shared application or workload identifier. Two costs outside that list cause, in our view, the most argument: evaluation runs, which spend tokens without serving a customer, and the central team's salaries.
Which allocation method fits each shared AI cost?
The FinOps Framework's Allocation capability names fixed, proportional and proxy-metric splits, and an even split, and says most organizations mix them, funding some shared items centrally. The methods for shared cloud costs generally are covered in shared cloud cost allocation; the table applies them to AI cost types and is our recommendation, not a standard.
| Cost | What drives it | Basis we'd use | Where the data comes from |
|---|---|---|---|
| Pay-as-you-go tokens | Requests, prompt size, model choice | Direct metering | Provider usage and cost APIs, gateway logs, CUR |
| Reserved capacity (PTUs, provisioned throughput, priority commitments) | A fixed block bought in advance | Share of traffic served on the reservation, with idle time charged to the teams that asked for it | Gateway logs with timestamps |
| Evaluation runs | Releases and model changes | Direct by eval set; cross-team model benchmarks split evenly or held centrally | Request tags on eval traffic |
| Vector databases and embeddings | Stored data and queries | Proxy metric: index size or query count | Database metrics |
| Gateway and observability | Request volume | Proportional to requests | Gateway logs |
| Central AI team | Roadmap and onboarding | Fixed share per onboarded production application; new-capability work held centrally | Headcount plan |
The consumption basis for reserved capacity follows the FinOps Foundation's paper on building a generative AI cost tracker. It notes that dividing a shared PTU's cost evenly across use cases is the simplest method and unfair when usage differs, and that combining utilization with timestamps gives a fair basis for showback or chargeback. Billing data won't always split reserved capacity: Anthropic's cost data shows priority traffic at zero, though its usage endpoint counts the tokens, and Google drops request labels on Provisioned Throughput. Use usage counts or gateway logs. For break-even on those purchases, see AI capacity reservations vs cloud commitments.
What does a worked allocation look like?
These numbers are illustrative, not from any company. A central AI service costs $380,000 in a month. Three teams consume it: a customer support assistant (Support), a sales copilot (Sales) and a code review agent (Engineering).
- Tokens, $180,000. Gateway metadata attributes $90,000 to Support, $54,000 to Sales and $27,000 to Engineering. $9,000 has no valid tag.
- Reserved capacity, $40,000. Support sends 75% of the traffic that runs on it and Sales 25%. Engineering runs batch jobs on pay-as-you-go and never touches it.
- Evaluation, $25,000. Team eval sets cost $8,000, $6,000 and $5,000. A $6,000 benchmark of a new model serves all three.
- Vector database, $20,000. Index size splits 50%, 40% and 10%.
- Gateway and observability, $15,000. Requests split 50%, 30% and 20%.
- Central AI team, $100,000. $60,000 supports four production applications (two for Support, one each for the others); $40,000 builds capabilities no team uses yet.
| Support | Sales | Engineering | Held centrally | Total | |
|---|---|---|---|---|---|
| Tokens | $90,000 | $54,000 | $27,000 | $9,000 | $180,000 |
| Reserved capacity | $30,000 | $10,000 | $0 | $0 | $40,000 |
| Evaluation | $10,000 | $8,000 | $7,000 | $0 | $25,000 |
| Vector database | $10,000 | $8,000 | $2,000 | $0 | $20,000 |
| Gateway and observability | $7,500 | $4,500 | $3,000 | $0 | $15,000 |
| Central AI team | $30,000 | $15,000 | $15,000 | $40,000 | $100,000 |
| Allocated | $177,500 | $99,500 | $54,000 | $49,000 | $380,000 |
| Shortcut: all cost pro rata to tagged tokens | $200,000 | $120,000 | $60,000 | $0 | $380,000 |
The shortcut divides $380,000 in proportion to the $171,000 of tagged token spend. It charges Support $22,500 more, Sales $20,500 more and Engineering $6,000 more. The net gap is exactly the $49,000 we hold centrally. The shortcut also shifts reserved capacity between teams: Engineering pays about $6,300 it never used and Support about $9,000 too little.
Both central holds are opinions. Untagged spend stays central as a data-quality figure, because spreading it removes the reason to fix tags. New-capability work stays off current bills, because charging it taxes the early adopters a central service most needs.
Should you show teams these costs or charge them?
Show first. The FinOps Foundation's token cost paper puts showback to application and team owners at its Walk stage and chargeback at Run, where organizational maturity supports it. Its overview adds that visibility alone often changes behavior. The move between the two is covered in showback to chargeback; for AI, we think the test is whether untagged spend stays under about 5% of token spend for two or three months running.
The FinOps Foundation says FOCUS 1.5, scheduled for ratification in December 2026, is set to add model identity, cache status and requester attribution to cost data. Until providers publish it, the gateway log remains the record of who used what.
An allocation rule holds only if each cost line links to the application it serves and the owner who answers for it, which is why we think AI spend belongs in the same operating model as the rest of the portfolio.
Sources
- FinOps Foundation, Managing AI Token Costs from Direct Model Provider APIs, last updated October 28, 2026
- FinOps Foundation, FinOps for AI Overview, last updated October 28, 2026
- FinOps Foundation, How to Build a Generative AI Cost and Usage Tracker, accessed November 22, 2026
- FinOps Foundation, Allocation capability, FinOps Framework, accessed November 22, 2026
- OpenAI, Costs endpoint, API reference and Usage, API reference, accessed November 22, 2026
- Anthropic, Usage and Cost Analytics API, accessed November 22, 2026
- AWS, Track usage and costs in Amazon Bedrock, Best practices for cost attribution, Application inference profiles and Frequently asked questions, accessed November 22, 2026
- Microsoft Learn, Cost and FinOps in Microsoft Foundry, updated November 17, 2026
- Microsoft Learn, Emit metrics for consumption of large language model tokens, updated September 15, 2026
- Google Cloud, Custom metadata labels, updated November 20, 2026
Frequently asked questions
Can you allocate AI token costs without a gateway?
Yes, for a single provider. Map each API key, project or workspace to one team and application, then use the provider's own reporting: OpenAI's Costs endpoint, Anthropic's Usage and Cost Analytics API, Bedrock application inference profiles or Foundry deployment tags. A gateway becomes necessary for several providers, feature-level attribution or policy on every request, which is where the FinOps Foundation recommends one.
Why does Bedrock show all our spend under one role?
When a gateway or proxy calls Amazon Bedrock for many callers, Bedrock records the gateway's IAM role as the caller. AWS suggests two fixes. For per-team dollars in billing, the gateway assumes its Bedrock role per tenant with session tags, caching credentials. For per-request detail, it sets request metadata on each call, which lands in invocation logs.
How should evaluation costs be allocated?
In our view, charge each team's own eval set to that team by tagging eval traffic, since releases and model changes drive it. Benchmarks that test a new model for everyone are better split evenly or funded centrally. Keep eval traffic on its own keys or tags so it doesn't inflate production cost per request or hide inside a team's token bill.
Should a shared PTU be split evenly across teams?
Only as a starting point. The FinOps Foundation calls an even split the simplest method but unfair when usage differs, and suggests allocating by consumption using utilization with timestamps. We would also charge idle reserved capacity to the teams that asked for the reservation, since their latency or throughput need is the reason it was bought.
Steve Elliott
Published November 22, 2026
More on AI Economics
Tokens, inference cost, cost per task.
Keep reading
Thinking tokens: how reasoning models bill for thought, and when it pays
Thinking tokens are billed as output, often unseen, and vary by request. How OpenAI, Anthropic and Google bill and cap them, with cost math on current prices.
Who Does Product Marketing Report To? Understanding Organizational Structure
Product marketing most often reports to marketing (the CMO), but sometimes to product management or the CEO. The right line depends on company size, industry and strategic goals.
Why AI business cases fail the CFO's test, and what a fundable one contains
Most AI business cases count hours, skip the baseline and leave out run costs. What finance looks for, a one-page template and a worked full-cost example.
Hours saved aren't dollars saved: turning AI productivity into financial value
AI time savings become money only when someone acts on them. Four routes that reach the P&L, how to measure each, and a worked example of the gap.