Self-hosting an open-weight model vs paying per token: where the break-even sits

In brief

At Google Cloud prices as of November 2026, self-hosting gpt-oss-120b struggles to beat the same model sold per token. The break-even math, and when it wins.

Steve Elliott

Steve Elliott · · 7 min read


On this page
  1. What are you actually comparing?
  2. What does a self-hosted token cost?
  3. Where does the break-even sit at different daily volumes?
  4. When does self-hosting win on cost?
  5. What do privacy and control change?
  6. How should you run the comparison for your own workload?
  7. Sources

If the open-weight model you want is also sold per token, our analysis says self-hosting it rarely saves money. On Google Cloud's prices as of November 2026, gpt-oss-120b on reserved B200 GPUs costs about what Google charges for the same model per token once the GPUs average a quarter of their benchmark throughput, and that is before staff and a second machine for redundancy.

The break-even moves a long way when the comparison is a proprietary API instead. Against a model priced about 20 times higher per token, a self-hosted LLM breaks even at about 1.3 billion tokens a day on two 8-GPU B200 machines, and lower on smaller ones that carry the load, but only if the open model does the job as well. In our view, that makes self-hosting a model-choice decision first and an infrastructure decision second.

What are you actually comparing?

There are three options: a proprietary model by API, an open-weight model by API (Google sells these as hosted endpoints, and Amazon Bedrock lists gpt-oss too), and the same open model on GPUs you pay for by the hour. Only the last two run the same model, so only that pair is a clean cost comparison. Proprietary against self-hosted is also a quality comparison, and the cost answer is worthless if the open model fails your evaluations.

Google's generative AI pricing page, accessed November 22, 2026, lists these rates per million tokens. The blended column is our arithmetic, for example (4 × $0.09 + $0.36) ÷ 5 = $0.144:

Model on Google CloudInputOutputBlended, 4 input tokens per output token
gpt-oss-120b$0.09$0.36$0.144
Llama 3.3 70B$0.72$0.72$0.72
Gemini 3.5 Flash (proprietary, global endpoint)$1.50$9.00$3.00

Batch rates are half for gpt-oss-120b. Newer Gemini Flash models list $2.70 blended from January 2027.

What does a self-hosted token cost?

Cost per million tokens is the machine's hourly price divided by the tokens it produces in that hour: its tokens per second times 3,600 times the share of the hour it is busy. Only the price is fixed; the rest depends on your workload, so a benchmark is a ceiling.

Google's per-VM prices for its 8-GPU machines, including vCPUs, memory and local SSD, per hour:

Machine typeGPUsOn-demandSpotDWS Flex-start1-year commitment3-year commitment
a3-highgpu-8g8 × H100 80GB$88.49$52.70$38.32$61.38$38.86
a3-ultragpu-8g8 × H200$84.81$50.87$42.40$58.47$37.21
a4-highgpu-8g8 × B200No rate listed$43.60$64.44$88.93$56.71

Spot capacity can be preempted, and Google says Spot prices can change up to once a day; DWS prices are variable but updated infrequently. Neither needs a commitment.

In our view, MLPerf Inference is the most carefully methodized public throughput figure. NVIDIA reports that an 8-GPU B200 system (CoreWeave's B200 SXM) in the v6.1 Closed Division served gpt-oss-120b at 90,205 tokens per second in the server scenario. That scenario allows 3 seconds to first token and 80 milliseconds per output token at the 99th percentile: about 12 tokens a second per stream, fine for batch and back-office work but slow for chat. In NVIDIA's GB200 NVL72 results, from different submitters, the 20 ms interactive scenario delivers about 69% of the server rate.

MLCommons gives the benchmark a mean input of 5,000 tokens and mean output of 1,250. The pages we read don't say whether the rate counts input. AMD's reproduction on eight MI355X GPUs reports 95,004 tokens per second at 69.17 samples per second offline, and 82,136 at 59.86 in the server scenario: about 1,370 tokens per sample either way, close to the output length. So we treat the figure as output tokens, each carrying about four input tokens, as in our blend. At the measured 3.6:1, the break-evens barely move.

At that rate, a comparable 8-GPU B200 VM such as the a4-highgpu-8g would produce about 325 million output tokens an hour, or about 1.6 billion tokens counting input. Hardware cost per million blended tokens:

Share of benchmark throughput, averaged over the monthSpot3-year commitment1-year commitmentGoogle API, standardGoogle API, batch
100%$0.027$0.035$0.055$0.144$0.072
50%$0.054$0.070$0.110$0.144$0.072
25%$0.107$0.140$0.219$0.144$0.072
10%$0.269$0.349$0.548$0.144$0.072

For the 3-year row at 100%: $56.71 ÷ 1,624 million tokens = $0.035. Hardware alone breaks even with Google's standard price at about 24% of benchmark throughput on a 3-year term, 38% on 1 year and 19% at the listed Spot price. Against batch pricing, those rise to about 49%, 76% and 37%.

The share is two discounts multiplied: how close your stack gets to a tuned benchmark run, and GPU utilization across 730 hours. Half the benchmark rate while busy, busy half the time, is 25%.

Where does the break-even sit at different daily volumes?

This illustrative case adds two costs that per-token pricing already covers:

  • Redundancy. At least two VMs, so one failure or upgrade doesn't take the service down.
  • People. Two engineers for serving, upgrades and on-call. The BLS puts median software developer pay at $135,980 in May 2025, and wages are 70.0% of private-industry compensation in its June 2026 data. That implies about $194,000 a year including benefits, or $32,400 a month for two: conservative for specialist ML infrastructure staff.

The table uses 3-year A4 commitments ($41,396 per VM a month) and Google's $0.144 rate. The planning case assumes 25% of benchmark throughput on average; the strong case, 60%, suits steady, batch-like traffic. We assume list prices on both sides; at these volumes, quotas and negotiated discounts come into play.

Tokens per dayGoogle API per monthSelf-hosted, planning caseSelf-hosted, strong case
1 billion$4,380$115,200 (2 VMs)$115,200 (2 VMs)
5 billion$21,900$115,200 (2 VMs)$115,200 (2 VMs)
20 billion$87,600$156,600 (3 VMs)$115,200 (2 VMs)
50 billion$219,000$280,800 (6 VMs)$156,600 (3 VMs)
100 billion$438,000$487,700 (11 VMs)$239,400 (5 VMs)

In the planning case, self-hosting barely catches up: each VM costs only about 3% less than the API would charge for the tokens it serves, so break-even comes near 250 billion tokens a day, and only if staffing doesn't grow. In the strong case, it is about 26 billion tokens a day.

The two-VM floor is a sizing choice, not a finding. OpenAI says gpt-oss-120b runs on "a single 80 GB GPU", and Google lists a single-GPU G4 VM (one RTX PRO 6000) at $1.98 an hour on a 3-year term. A pair costs about $2,900 a month, over 90% less than two B200 machines, leaving staff as the main fixed cost. No public MLPerf figure covers that machine; load test first.

When does self-hosting win on cost?

We think the cost case holds in four situations:

  • The alternative is a proprietary API. Against Gemini 3.5 Flash at a blended $3.00, two B200 VMs and two engineers break even at about 1.3 billion tokens a day. With two right-sized single-GPU VMs, if they carry the load, it is about 0.4 billion. Either way, your evaluations must show the open model meets the quality bar.
  • Traffic is steady or can wait. Overnight document processing and backfills keep GPUs busy and can run on Spot or Flex-start capacity.
  • The capacity is already paid for. If GPUs sit under an existing commitment or on premises, the marginal cost of tokens is much lower. See AI capacity reservations vs cloud commitments and cloud repatriation economics.
  • The model is small or tuned. OpenAI says gpt-oss-20b can run with 16 GB of memory. A fine-tuned model that no one sells per token has no API price to compare against.

Each still needs the full total cost of ownership, and tokens are usually a minority of AI initiative cost anyway, as we set out in AI initiative costs: tokens vs labor.

What do privacy and control change?

They are real reasons to self-host, but not cost savings; in our view a business case should price them separately. Self-hosting keeps prompts inside your network and lets you pin model versions and fine-tune freely. A hosted open model in your existing cloud account may meet some of the same needs; check the provider's data terms.

Read the license, too. gpt-oss is released under Apache 2.0. Llama 3.3's community license requires a "Built with Llama" notice if you distribute it or a product built on it, and incorporates Meta's acceptable use policy. Services above 700 million monthly actives in the month before its December 6, 2024 release must ask Meta for a license. This is not legal advice; confirm license terms with your counsel.

How should you run the comparison for your own workload?

  1. Measure the token mix per request from logs.
  2. Price the same model by API first, including its batch rate.
  3. Load test at your latency target. MLPerf shows the ceiling, not your number.
  4. Model the hours, not the peak. Average utilization across 730 hours decides the answer.
  5. Size the machine, then add redundancy, people and commitment risk. Price the smallest machine that carries the load, at least two replicas, and the term you would sign.
  6. Price privacy and control separately.

A self-hosting decision commits money for years against a forecast of volume and model quality. In our view, the commitment should be recorded next to the service it funds and the person accountable for its cost, in the same operating model that tracks both.

Sources

Frequently asked questions

At what volume does self-hosting an LLM become cheaper than an API?

It depends on the API it replaces. Against Google's $0.144 blended rate for gpt-oss-120b, our illustrative case breaks even near 26 billion tokens a day at 60% of benchmark throughput, and only near 250 billion at 25%. Against a proprietary model at a blended $3.00, it breaks even near 1.3 billion a day on two B200 machines, and lower on smaller ones, if the open model is good enough.

Why are hosted open-weight models so cheap per token?

We think it is because providers pool many customers' traffic and keep GPUs busy, which a single enterprise struggles to match. On Google Cloud as of November 2026, gpt-oss-120b costs $0.09 per million input tokens and $0.36 per million output tokens. In our worked example, a reserved 8-GPU B200 machine only matches that price when it averages about a quarter of its benchmark throughput.

Can I use MLPerf results to size a self-hosted deployment?

Use them as a ceiling. MLPerf runs are tuned and use a set dataset; NVIDIA reports 90,205 tokens per second for gpt-oss-120b on 8 B200 GPUs at a loose 80 ms per output token. Your prompts, latency target and traffic pattern will produce a lower rate. Load test your own workload on your serving stack before committing to GPUs for a year or more.

Should privacy requirements decide self-hosting?

They can, but treat them as a separate requirement with its own price. Self-hosting keeps prompts in your network and lets you pin model versions. In our view, folding privacy into a token cost comparison hides what you are paying for it. Also check whether a hosted open model in your existing cloud account already meets your data terms.

Free playbook

Run the enterprise from any AI.

Read The Headless Enterprise
Steve Elliott

Steve Elliott

Published November 22, 2026

More on AI Economics

Tokens, inference cost, cost per task.