Take one 8x H100 node at current on demand neocloud pricing, around 2.60 dollars per GPU hour, so about 20.80 dollars an hour for the node. Serve one model on it. Depending on nothing but how you configure and feed that node, your cost per million tokens lands somewhere between roughly 36 cents and 12 dollars.
That is a 32 times spread on identical hardware. No new GPUs, no different model, no vendor change. Just throughput, utilization, workload shape, and how much work you avoid doing twice.
This is what token economics actually is. Not a pricing page, not a vendor comparison, but understanding which variables in your own stack move your unit cost and by how much. And the reason most teams cannot answer "what does a million tokens cost us" is not that the math is hard. It is that they are not measuring the denominators.
This post covers the formula, every factor that moves it, worked numbers you can check, and then the part most people actually need: the specific metrics to put on a dashboard so you can see where the money goes.
The Only Formula You Need
For self hosted inference, cost per million tokens reduces to one expression:
Cost per Mtok = (node cost per hour) / (tokens per second × 3600) × 1,000,000
Use the full node or pod cost in the numerator, not the per GPU cost, whenever the model spans multiple GPUs. If you need four GPUs to hold the weights, all four are on the bill whether or not the fourth is doing much.
Three terms, and only three. Everything in this post is ultimately a way of moving one of them:
- Node cost per hour. What you pay for the hardware.
- Tokens per second. What you get out of it.
- Utilization. The hidden third term, because the formula above assumes the node is busy. If it is idle half the time, your real cost doubles.
Here is the second term's sensitivity on that 20.80 dollar per hour node, at full utilization:
| Aggregate throughput | Cost per million tokens |
|---|---|
| 500 tok/s | $11.56 |
| 1,000 tok/s | $5.78 |
| 2,000 tok/s | $2.89 |
| 4,000 tok/s | $1.44 |
| 8,000 tok/s | $0.72 |
| 16,000 tok/s | $0.36 |
Throughput is not a performance metric. It is a price.
The Cost Driver Tree
That fourth branch, work avoided, is the one most cost reviews miss entirely, and on agentic workloads it is the largest single lever. More on that below.
Factor 1: The Price of Compute
The least interesting lever, but the one everyone reaches for first.
Current on demand rates from an October 2026 pricing survey give a sense of the spread between GPU generations on the same provider: H100 SXM 80 GB around 2.60 dollars per hour, H200 SXM 141 GB around 3.59, B200 180 GB around 5.50, B300 288 GB around 6.94. The same survey shows a different provider charging between about 2% and 52% more for the same silicon, which tells you provider choice is worth as much as GPU choice in some cases.
What matters more than the headline rate:
Newer and more expensive is often cheaper per token. A B200 at twice the hourly rate of an H100 is a cost win if it delivers more than twice the throughput on your model, which for large models with FP4 support it frequently does. Never compare GPUs on hourly price. Compare them on cost per million tokens for your model at your batch size.
Commitment versus flexibility is a real tradeoff. Reserved pricing for multi month commitments is quoted separately and is meaningfully cheaper, but it converts a variable cost into a fixed one. That is only a win if your utilization is high, which loops back to the third term.
Spot capacity is cheap if your workload checkpoints. For batch inference, offline evaluation, and embedding generation, spot is close to free money. For interactive serving it is a reliability problem wearing a discount.
Do not forget the bill around the bill. Egress, NAT gateways, load balancers, and storage are real line items. At roughly 9 cents per GB outbound on a major cloud, 10 TB of monthly egress is about 900 dollars, which can be a double digit percentage of a small deployment's total.
Factor 2: Throughput
This is where most of the engineering leverage lives, and it is why the same hardware spans a 32 times cost range.
Model size and active parameters. Obvious for dense models. Less obvious for mixture of experts, where what matters is active parameters per token, not total. A model with 600B total and 37B active behaves, for throughput purposes, far more like a 37B than a 600B, while still needing memory for all the weights. This is why MoE has become the dominant frontier architecture: it decouples capability from per token compute cost.
Numeric precision. Weight and KV cache quantization both raise throughput, partly by reducing memory traffic and partly by enabling larger batches. Reported gains in the range of 1.3 to 2 times for FP8 over FP16 on H100, and another 1.5 to 2 times for FP4 over FP8 on B200, are typical of what vendors and FinOps writeups claim. Validate on your own model and evals, because the accuracy cost varies by task, but the direction is reliable and the magnitude is large.
Batching strategy. This is the single biggest configuration difference between a cheap deployment and an expensive one. Static batching typically leaves GPU utilization in the 15 to 30 percent range. Continuous batching, which every modern engine implements, moves it to roughly 60 to 80 percent. That is a three to four times effective throughput gain at identical hardware cost, which in the table above is the difference between 2.89 and 0.72 dollars per million tokens.
Speculative decoding. A small draft model proposes several tokens, the large model verifies them in one pass. For output heavy workloads the reported range is roughly 2 to 4 times improvement in decode throughput. The caveat is acceptance rate: when draft tokens are usually rejected, you pay the draft cost for nothing. Measure acceptance rate, not just the headline speedup.
Distillation and model routing. If a 14B model handles 70 percent of your traffic at acceptable quality, routing that traffic away from a 70B is a four to eight times cost reduction on the majority of your volume. This is usually the highest leverage change available, and it is a product decision as much as an infrastructure one.
Factor 3: Utilization, the Silent Multiplier
The formula assumes a busy node. Reality rarely cooperates. Here is the same 4,000 tokens per second node at different utilization levels:
| Utilization | Effective cost per Mtok | Waste versus full utilization |
|---|---|---|
| 15% | $9.63 | $8.19 |
| 30% | $4.81 | $3.37 |
| 50% | $2.89 | $1.44 |
| 70% | $2.06 | $0.62 |
| 90% | $1.60 | $0.16 |
A node running at 30 percent utilization costs 3.3 times more per token than the same node at 90 percent. No amount of quantization or kernel optimization recovers that, because the GPU is not slow, it is empty.
Where utilization goes to die:
- Provisioning for peak. Interactive traffic has a daily shape. Capacity sized for the 2pm peak sits idle at 4am.
- Cold starts preventing scale down. If a replica takes ten minutes to load weights and warm up, you cannot scale down aggressively, so you do not scale down at all.
- Per team or per model dedicated capacity. Five teams each with their own underutilized endpoint is five times the waste of one shared pool with proper isolation.
- Reserved capacity bought for a forecast that did not arrive.
The fixes are mundane and effective: consolidate onto shared pools, mix batch workloads into interactive capacity troughs, use spot for the batch portion, and keep a small warm pool rather than scaling to zero.
Factor 4: Workload Shape and the Prefill Decode Asymmetry
Here is something that trips up almost every cost model: an input token and an output token do not cost the same.
Prefill processes all input tokens in parallel and is compute bound, so it achieves high aggregate throughput. Decode generates one token at a time per sequence and is memory bandwidth bound, so its throughput is much lower. Using plausible figures for a 70B class model on our node, say 20,000 tokens per second for prefill and 4,000 for decode:
| Token type | Cost per million |
|---|---|
| Input (prefill) | $0.29 |
| Output (decode) | $1.44 |
A five times ratio, which is why commercial API pricing almost always charges more for output tokens than input, usually by a similar multiple. If your internal cost model uses a single blended rate, it will systematically misprice every workload: it overcharges summarization, which is input heavy, and undercharges generation, which is output heavy.
Workload shape therefore changes your cost per token even with everything else fixed:
- Classification and extraction. Huge input, tiny output. Cheap per token, and dominated by prefill, so prefill throughput and caching are what matter.
- Chat. Roughly balanced, moderate context.
- Long form generation. Small input, large output. Expensive per token, and decode bound, so this is where speculative decoding and memory bandwidth earn their keep.
- Agentic and coding loops. Extreme input heaviness. Real coding agent traces show input to output ratios around 131 to 1, with context reaching about 80K tokens by turn 30. Almost all the compute is prefill.
Factor 5: Work Avoided, the Biggest Lever Nobody Models
If prefill dominates your compute and most of your prefill is a prefix you already computed, then your cost is mostly determined by your cache hit rate. Not your GPU, not your quantization, your cache hit rate.
Work the numbers on one agentic turn: 80,000 input tokens, 600 output tokens, on our node with 20,000 tok/s prefill and 4,000 tok/s decode.
| Prefix cache hit rate | Prefill time | Decode time | Node time per turn | Relative cost |
|---|---|---|---|---|
| 0% | 4.00s | 0.15s | 4.15s | 1.0x |
| 60% | 1.60s | 0.15s | 1.75s | 2.4x cheaper |
| 94% | 0.24s | 0.15s | 0.39s | 10.6x cheaper |
At zero hit rate, prefill is 96 percent of the node time for that turn. The decode you actually wanted, those 600 tokens, is a rounding error. You are paying almost entirely to re read a conversation you already read.
Extend it to a full 33 turn session with context growing each turn, and the same node goes from about 64 node seconds per session to about 8.5, which is roughly 37 cents versus 5 cents per session at our node price. Multiply by your session volume and this is usually the largest number on the page.
The operational catch, which I have written about separately in more depth: a prefix cache only helps if the request lands on the instance holding the cache. Published results from shared cache pool work show hit rate going from 1.7 percent to 92.2 percent on a real agent trace workload with no change to the model or engine, purely from where the cache lived and how requests were routed. Enabling prefix caching is a config flag. Actually achieving a high hit rate is a routing and topology problem.
Other forms of avoidable work worth measuring:
Retries. A request that fails after generating 400 tokens and gets retried costs you 400 tokens of pure waste, and nothing in your token counter distinguishes it from useful output unless you label it.
Preemption recompute. When KV cache runs out, engines preempt and later recompute requests. Every preemption is work thrown away. It shows up as throughput loss, not as an error.
Thinking tokens. Reasoning models generate a long internal trace before the answer. A task that produces 200 useful tokens might generate 2,000 total. You pay for all of them. That can be entirely worth it for hard problems and pure waste on easy ones, which is why routing by difficulty matters.
Multi agent fan out. Published engineering accounts put multi agent systems at roughly 15 times the token consumption of a chat interaction, against about 4 times for a single agent. Fan out is a legitimate technique with a large and specific price tag.
Prompt bloat. Few shot examples nobody validated, instructions duplicated across layers, retrieved context padded to a round number. At 131 to 1 input ratios, prompt engineering is cost engineering.
Factor 6: Power
Increasingly the binding constraint rather than a cost line. If your organization operates its own capacity, the metrics that matter are tokens per megawatt and joules per token, and they are now standard in serious inference benchmarking alongside tokens per second and cost per million tokens. The reason is simple: when you cannot get more power allocated, the only way to serve more tokens is to serve more tokens per watt. Infrastructure programs built around maximizing token performance per megawatt exist precisely because of this ceiling.
The Metrics to Track
This is the part you can act on tomorrow. Organize it in five tiers, because different audiences need different levels.
Tier 1: Unit economics, the numbers you report upward
| Metric | Definition | Why it matters |
|---|---|---|
| Cost per million input tokens | Compute cost attributed to prefill divided by input token volume | Separates the two sides of the asymmetry |
| Cost per million output tokens | Compute cost attributed to decode divided by output token volume | The expensive side, and the one that scales with usage |
| Blended cost per Mtok | Total compute cost divided by total tokens | Useful for trending, dangerous for decisions |
| Cost per request | Total cost divided by request count | Connects infra to product behavior |
| Cost per session or task | Cost across all turns of a user session or agent run | The only honest unit for agentic and multi turn products |
| Cost per successful outcome | Cost divided by tasks that actually succeeded | The number that includes your retry and failure waste |
That last one is the metric most worth adding and least commonly present. If 30 percent of your agent runs fail and get retried, your cost per successful outcome is 1.4 times your cost per run, and only one of those numbers reflects reality.
Tier 2: Throughput and efficiency, the engineering levers
| Metric | Why it matters |
|---|---|
| Output tokens per second per GPU | The denominator in the cost formula, normalized so you can compare hardware |
| Total tokens per GPU hour | Converts directly to cost per token at a known hourly rate |
| GPU utilization (SM occupancy) | The utilization multiplier. Under 50 percent is recoverable money |
| Model FLOPs utilization | How much of the GPU's theoretical capability you are actually using |
| Batch size, running and max | Low running batch with available capacity means you are latency limited, not compute limited |
| KV cache utilization | Sustained above 90 percent predicts preemption |
| Prefix cache hit rate | On input heavy workloads, the single strongest cost predictor |
| Preemption count | Each one is discarded work |
| Queue depth | Rising queue with high utilization means add capacity, rising queue with low utilization means fix scheduling |
| Speculative decode acceptance rate | Below roughly 50 percent and the technique may be costing more than it saves |
| Tokens per megawatt, joules per token | Where power is the constraint |
Tier 3: Latency SLOs, the constraint on everything above
You cannot optimize cost without bounding latency, because the cheapest configuration is always "batch everything and make users wait."
| Metric | Why it matters |
|---|---|
| TTFT p50 and p99 | The latency users feel first, and what caching improves |
| TPOT or inter token latency | Perceived streaming speed, degrades as batch size grows |
| End to end latency per request | The contract with the product |
| TTFT split by cache hit versus miss | Shows what reuse actually buys, which a blended average hides |
| Queue wait time | The portion of latency that is purely capacity |
Track the pair, always. Cost per token and p99 TTFT on the same dashboard, because improving one at the expense of the other is not an optimization, it is a trade you should be making deliberately.
Tier 4: Waste, the category nobody instruments
| Metric | Why it matters |
|---|---|
| Retry rate and tokens burned on retries | Pure waste, invisible in standard token counts |
| Finish reason distribution | A high rate of length truncation means you are paying for output that gets cut off |
| Tokens discarded by preemption recompute | Work done twice |
| Thinking token share of total output | Tells you whether reasoning mode is earning its cost |
| Idle GPU hours | Capacity paid for and not used |
| Cache miss recompute volume | Prefill tokens you could have skipped |
| Prompt token share that is static boilerplate | Your caching and prompt trimming opportunity, quantified |
Tier 5: Business framing, for the conversation outside engineering
| Metric | Why it matters |
|---|---|
| Cost per active user per month | The number that goes in a pricing model |
| Inference cost as a percentage of revenue | Whether the product works as a business |
| Gross margin per request or per plan tier | Where you are losing money by customer segment |
| Cost trend versus volume trend | Whether your efficiency work is outrunning your growth |
That last one is the health check. If token volume is growing 3 times year over year and cost is growing 3 times, your efficiency work is doing nothing. Industry telemetry has put median enterprise monthly LLM spend growth around 7 times year over year, which means plenty of organizations are in exactly that position.
How to Instrument It
Three things make or break this:
Label everything at the request level. Team, product surface, model, use case, and for agentic systems a run identifier that ties an orchestrator and all its workers into one trace. Without labels you have a total cost and no way to attribute it, which means no way to decide what to fix.
Use the OpenTelemetry GenAI semantic conventions for the application layer rather than inventing your own schema. They define spans for model calls with token usage attributes and spans for agent operations, which is what lets you compute cost per session across a multi step agent run rather than per API call. One caveat worth knowing: these conventions have been in development rather than stable status, so pin your version and expect migrations.
Put token counts next to GPU spend in the same place. The most common failure mode in LLM cost work is that spend lives in the billing console and tokens live in application logs, so nobody can divide one by the other. Cost per million tokens is a ratio, and you need both halves in one system to compute it weekly.
Levers, Ranked
Rough ordering by return on effort for a typical self hosted deployment:
| Lever | Typical impact | Effort | Notes |
|---|---|---|---|
| Continuous batching | 3 to 4x | Low | Usually already on in modern engines, verify it is |
| Prefix caching with correct routing | Up to 10x on input heavy workloads | Medium | Routing is the hard part, not the flag |
| FP8 KV cache and weights | 1.3 to 2x | Low | Needs an eval run to confirm quality |
| Right size the model, route by difficulty | 4 to 8x on routed traffic | Medium | Product and infra decision together |
| Raise utilization by consolidating | Up to 3x | Medium | Organizational more than technical |
| Trim prompts and context | Proportional to input share | Low | Highest return on agentic workloads |
| Reserved or spot pricing | 30 to 60% on the eligible portion | Low | Requires predictable load or checkpointing |
| Speculative decoding | 2 to 4x on decode | Medium | Watch acceptance rate |
| Newer GPU generation | Varies, often favorable | High | Compare on cost per token, never per hour |
| Prefill and decode disaggregation | Latency, not throughput | High | Do not expect a cost win from this alone |
Start at the top. Teams routinely jump to the bottom half of this table while leaving a three times batching win or a ten times caching win on the table.
A Note on Self Hosting Versus APIs
The break even depends almost entirely on utilization, which is the point of this whole post. Self hosting replaces a per token price with a fixed hourly cost, so it wins only when you keep the hardware busy. A node at 15 percent utilization is usually more expensive than an API, and the same node at 85 percent is usually dramatically cheaper.
FinOps writeups commonly place the crossover somewhere in the range of tens of millions of tokens per month, but treat any specific number with suspicion. It depends on your model, your workload shape, your utilization, and whichever API price is current this week, and API prices have been falling steadily and fast. The durable version of the analysis is the formula at the top of this post: compute your real cost per million tokens at your real utilization, then compare it against today's list price. Recompute quarterly, because both sides move.
Pitfalls
- Using one blended token price. It misprices every workload because input and output costs differ by roughly five times.
- Reporting cost per token without cost per outcome. Hides all your retry and failure waste.
- Comparing GPUs on hourly price. The only valid comparison is cost per million tokens on your model.
- Measuring throughput at batch size one. It tells you nothing about serving economics.
- Benchmarking with unique synthetic prompts. It measures the one scenario where caching cannot help, which is the opposite of production.
- Ignoring utilization in the cost model. A 30 percent utilized node costs 3.3 times its nameplate cost per token.
- Tracking tokens and spend in separate systems. Then the ratio nobody can compute is the only number that matters.
- Treating cost as an infrastructure problem only. Prompt size, model routing, retry policy, and whether reasoning mode is on are all product decisions with direct cost consequences.
- Optimizing cost without a latency SLO. The cheapest configuration always violates it.
Conclusion
Cost per token is not a property of your model or your GPU. It is a property of your whole system: what you pay for compute, how much throughput you extract, how busy you keep it, and how much repeated work you manage to skip. The 32 times spread at the top of this post is not hypothetical, it is the gap between a deployment that is tuned and one that is merely working.
If you are starting from nothing, do these three in order. Get token counts and GPU spend into the same system so you can compute cost per million tokens weekly, split by input and output. Check your utilization and your prefix cache hit rate, because those two numbers alone explain most unit cost gaps. Then add cost per successful outcome, so the retries and failures stop hiding inside your averages.
Everything else on this page is an optimization. Those three are visibility, and you cannot optimize a number you are not computing.
References
- GPU cloud pricing survey, October 2026: What an H100, H200, B200 or B300 costs per hour
- Spheron: AI Inference Cost Economics in 2026, GPU FinOps Playbook
- vLLM blog: Serving Agentic Workloads at Scale with vLLM and Mooncake
- Anthropic: How we built our multi-agent research system
- vLLM docs: Optimization and Tuning
- vLLM docs: Production Metrics
- SemiAnalysis: InferenceX, open source inference benchmarking
- MLCommons: MLPerf Inference v6.1 results
- OpenTelemetry: AI agent observability and GenAI semantic conventions