Frontier labs have made their most powerful models available via an API key, which is charged per million input and output tokens. OpenAI and Anthropic have been providing API access to their models since early 2023, and other frontier labs have been catching up since.

According to LLM Pricing website, in 2023, to access a frontier model, you would be paying ~$30/MTok. But by mid-2026, far more capable models are available at roughly 95% cheaper, sitting in the range of $0.50 to $3 per MTok.

I have always been curious about this process, and I write this blog to answer a simple question:

$$\textbf{What exactly determines a hosted model's pricing?}$$

Turns out the answer isn't very simple; it involves a set of engineering trade-offs and economic decisions that determine these prices.

Knowing this could help us answer other questions: what is batch processing, how are some models cheaper than others, what exactly is fast mode, and why is it more expensive?

We will explore this process by serving Qwen 3.8 27B, a dense model, on a single Nvidia RTX Pro 6000 Blackwell GPU.

The GPU rates

Let's start with our GPU: an Nvidia RTX Pro 6000 Blackwell. It features 24,064 CUDA cores, 752 Tensor cores, and 96GB of GDDR7 VRAM.

For the sake of this blog, we're only going to rent GPUs online. A single one of these costs roughly $1.5/hr.

Why just one? Our model is 27B parameters at bfloat16, which we'll see works out to about 56GB — comfortably inside 96GB of VRAM, with room left over for the KV cache. Reaching for more GPUs than the model needs would just be spending money on a problem we don't have. Multi-GPU setups exist for models too large to fit on one card, or for serving far more users at once than a single card can hold — and we'll get to exactly when that becomes necessary later on.

Buying the GPUs

I think it's a good idea to understand the difference between buying and renting a GPU. If you are a cloud provider like RunPod or Lambda Labs, you would typically buy these GPUs instead of renting them.

On the market, this GPU will cost you roughly $16,000 (August 2026) - that's pretty expensive!

For this example, we'll assume a three-year economic lifespan. A single year has 8760 hours, so amortising the purchase over 3 years:

$$\text{Amortized cost } \frac{\$16,000}{(3 \cdot 8760)\text{ hours}} = \$0.61/\text{hour}$$

Now this seems far cheaper than our rental rate of $1.5/hr, but this is just the capital expenditure. We haven't accounted for the Operational Expenditure (OpEx), which includes property lease, electricity, cooling, networking, and maintenance.

Let's say that after CapEx and OpEx together, the real cost comes to roughly $1/hr. This is still less than the $1.5/hr we're paying to rent. That $0.50 difference is what gives neoclouds room to cover their additional operating costs and earn a profit.

This being said, let's use our rental rate of $1.5/hr per GPU for further calculation of token pricing.

GPU hours to token pricing

Coming back to our model, Qwen 3.8 27B was released in August 2026. We're running the model at its native precision of bfloat16 (16 bits or 2 bytes). The exact parameter count is 27,781,429,744 (27.78 billion), and in native precision this would take 27.78 B * 2 bytes ≈ 55.56 GB of VRAM.

You would require almost 56GB of VRAM just to load the model. We haven't loaded the KV cache yet. After renting a single GPU on Vast.ai, I performed some benchmarks.

Getting a single-user price

With 1024 input tokens and 512 output tokens, the prefill and decode speeds were 3063.38 tok/s and 29.1 tok/s, respectively, with a total latency of 17.93s. The total latency here means the total time from when the user sends the request to getting the full response.

With the single GPU rent of $1.5/hr, we get the token pricing as follows:

$$\begin{align*} &\text{Input tokens } \frac{\$ 1.5}{3600s} \cdot \frac{1s}{3063.38 \text{ tok}} = 0.14 \times 10^{-6} \text{ \$/tok} = 0.14 \text{ \$/MTok} \\[1em] &\text{Output tokens } \frac{\$ 1.5}{3600s} \cdot \frac{1s}{29.1 \text{ tok}} = 0.00001432 \text{ \$/tok} = 14.32 \text{ \$/MTok} \\[1em] &\text{Total cost of request } \frac{\$ 1.5}{3600s} \times 17.93s = \$0.00747083 \end{align*}$$

The input is cheap, but the output is very expensive!

For our particular example, the total cost we paid was:

$$\frac{\$0.14}{10^6 \text{ toks}} \times 1024 \text{ input toks} + \frac{\$14.32}{10^6 \text{ toks}} \times 512 \text{ output toks} = \$0.0074752$$

This is roughly the same as the total cost of the request. But once we have more users, we will see that the per-user request cost would differ substantially.

Scaling up input

The rates that we just estimated are assumed only if the input and output tokens are 1K and 512, respectively. But would these rates hold up the same if we change input lengths?

If we double the input sequence length and keep the output length as 512, we see the pricing as follows:

Here, Cost/request means how much it costs for the provider to serve the request (GPU seconds x rental rate), and Billed at flat rate means what you, the user, pay for processing your tokens under a published per-token price.

The Billed at flat rate is calculated assuming the pricing is fixed at $0.136/Mtok for input and $14.32/Mtok for output.

We observe that for small and medium input lengths, the input pricing mostly remains the same and the cost of request remains roughly the same as what you pay.

At large context (input tokens), the prefill throughput degrades. The attention work grows with the increased input sequence length, and as we will see, the size of our KV-cache balloons drastically.

This is why input pricing is much higher at 256K than 1K. Had our pricing model assumed a fixed input rate of $0.136/Mtok, the provider would bill us $0.04 while spending $0.10 for the GPU = a loss $0.06. If you were to serve millions of requests daily, that small per-request loss adds up very quickly.

Now the provider has 3 choices:

1. Cap the context to a particular length, say 100K input tokens, to keep the Cost per request and token bill the same
2. Eat this loss of $0.06 and subsidise all long-context requests, like Anthropic.
3. Set up a new pricing tier for long-context requests, and charge roughly double the standard prices, like OpenAI, Google or xAI (part of SpaceX).

The growing KV-cache

KV-caching is basically a memory optimisation technique where the Key and the Value vectors generated by the Self-attention layers for past tokens are stored. Without it, the model would have to re-run the entire attention algorithm with the long input at every step.

The formula for finding the KV cache is as follows:

$$\text{KV cache } = 2 \times B \times L \times H_{kv} \times d_{head} \times \text{S} \times N$$

where:

• $B$ is the bytes per precision. For bfloat16, this would be 2 bytes; int8 = 1 byte, etc.
• $L$ is the number of full attention layers of the model (Qwen 3.8 27B has 16).
• $H_{kv}$ is the number of KV-heads of the model.
• $d_{head}$ is the head dimension of each attention head.

We previously saw that as the input length increases, the input latency also increases. From the formula, we can also see why the KV cache increases linearly.

The grey bar represents the size of the model weights (~56 GB). For shorter context lengths, the model weights take up most of the GPU memory. At larger contexts, the KV cache starts to take up a significant portion of memory along with the model.

Why are cache reads not free?

One would think that since the provider is saving on the compute, the price of cache reads should essentially be free. While caching may have saved us from prefill, the model still has to read from the KV cache at every step for decoding.

In our own specific example, we can see that out of 96 GB, we're using 72.78GB of VRAM alone. Roughly 76% of the memory is just for one single user - it can't be used to serve another user if we wanted to. And this is the reason why cache reads aren't free, because the provider is still charging you for storing your data on their cards.

An increase in KV cache would also mean slower token generation speeds. Because the model now has to read more bytes per decode step, the process increasingly gets memory-bound, which also results in increased total latency.

Some providers also charge for cache writes, at a rate slightly more than regular input pricing. The write still pays for full prefill, and the VRAM is held for a fixed TTL. The extra pricing from the input is for the memory that can't be used for any other request and is solely reserved for you during the TTL.

Serving multiple users

So far, our discussion has been serving just 1 user on a single GPU. We were able to obtain a cheap input price of ~$0.14/Mtok. But our output price was still quite high for a small model, at around $14.32/Mtok.

Also, our GPU compute was underutilised, since in the decoding phase the model has to read the entire model weights and the KV cache from the GPU memory to produce the next token. The GPU cores spend most of the time waiting on those memory reads rather than doing the math.

Increasing batches

What if instead of serving just one single user, we make this infrastructure available for more users? By introducing more users per served batch, we are making sure that our GPU is always fully utilised.

We can now load the model weights once and perform inference with different requests, and so serve multiple customers simultaneously. But this comes at a trade-off: serving more batches would mean a higher throughput, but each user now experiences a higher latency on their end, because their request is being processed on the GPU shared with multiple other requests.

I performed another benchmark on a single GPU with increasing batch sizes, computed the pricing and plotted the results as follows:

From this graph, we can see that on the same GPU, the output pricing goes from $14.34/Mtok B=1 to $0.65/Mtok B=160: a 22x drop in output pricing. We also see that as the number of batches increases, the per-user latency also increases significantly. By adding more batches, we're trading the output pricing for user latency.

The input pricing, however, remains the same at ~$0.14/Mtok. This is because prefill is compute-bound, and it processes the input tokens in parallel. For most input lengths, the GPU cores are already being fully utilised. Adding more batches would queue the processing work.

In our case, B=32 sits at the knee of the curve between user latency and output pricing. The per-user latency is 50.6 seconds, and the rates are $0.14/Mtok for input and $1.01/Mtok for output

But 50 seconds for a 512-token output is slow. Whether that's acceptable is more of a product decision than an engineering one. For an interactive chat, B=16 at 33.5s latency is more ideal, but the output now costs $1.43/Mtok. And if you're running a batch job where no one is immediately waiting for a quick response, B=160 becomes more ideal.

For the remainder of our blog, let's assume B=16 is ideal for us. This gives us a final pricing of $0.14/Mtok for input and $1.43/Mtok for output.

An explosion of KV cache

Previously, we computed the size of the KV cache for a single user $N=1$. Now, assuming that we have 16 users and each of those sends an input with a length of 128K, we see that the KV cache size increases dramatically.

Our model, Qwen 3.8 27B, supports a context window of 262,144 tokens. We can see that for one user, 1 GPU is more than enough. But now we would require more than 1 GPU connected and 'talking' to each other, so they behave as one large GPU, to serve multiple users.

Here's an exact table which tells us how many GPUs you would require to serve multiple users:

Note: These values are specific to Nvidia RTX Pro 6000 Blackwell. Furthermore, this is assuming that parallelism has virtually no overhead. In real systems, parallelism would add some overhead.

Key takeaways

Why bother packing more users onto a GPU at all? One user fits comfortably — even at full 256K context, we only used 76% of the card. The problem isn't space; it's that a single user leaves the GPU's compute mostly idle while it waits on memory reads. We're paying for a whole card and using a fraction of it.

It turns out batching actually makes this whole business of inferencing viable. This can enable the providers to slightly increase the measured token prices to a competitive market rate for a profit margin.

We earlier saw that for B=1 at 1024 input tokens and 512 output tokens, GPU costs us $0.00747, and the user is billed $0.00747 in tokens at 17.95s latency. The two match because the user had the entire GPU to themselves during this window.

At B=16, the per-user latency is now 33.5s and costs each user $0.00087 in tokens, but it served 16 users in that window (assuming each user sent 1024 in and 512 out tokens). The GPU cost for that 33.5s window is $0.01395.

The GPU cost is now split among 16 different users, reducing the per-request cost.

To actually make a profit, we can slightly increase the token cost from our derived $0.14/Mtok input and $1.43/Mtok output to $0.2/Mtok input and $2/Mtok output.

Here's how this scales up with 1024 input tokens and 512 output tokens; all are per window:

Somewhere at around B=10, serving the model becomes a profitable business. Below that, the provider loses money on every request.

And from our discussion, we can finally say that if we were to host our model, it should be priced at $0.2/Mtok input and $2/Mtok output.

This also explains why batch APIs are so much cheaper. If a request doesn't need an immediate response, the provider can hold it until they've accumulated enough work to run a very large batch - the flat part of our curve, where output costs $0.65/Mtok instead of $1.43. The 24-hour turnaround isn't a limitation, it's the whole product: you're trading latency for a seat in a much bigger batch.

The main point I want you to takeaway from our discussion is this: Having a sufficient demand and continuously serving big batches is the key to running a profitable LLM inference business; if you can't support large batches, your cost per token will balloon, making your operation unprofitable.

Fast and Ultrafast mode

We saw that at lower batch sizes, each request gets more of a 'slice' of the GPU with reduced latency and faster processing, but increased pricing. Rather than only serving large batches, frontier labs provide a "Fast mode" option, which is more expensive than the regular pricing.

Here's a table showing Fast mode pricing as before:

For most of the requests, fast mode is a feature you wouldn't require and the standard pricing can do the job. From the provider's perspective, if the demand is relatively less, a lower batch with premium pricing adds another stream of revenue.

Our entire discussion has assumed that we're using an Nvidia GPU for inference. But what if we design a chip that is dedicated for LLM inference?

At the time of writing this blog in August 2026, OpenAI announced Ultrafast mode on their platform. They partnered with Cerebras that develops chips dedicated for inference which features keeping the weights in on-chip SRAM rather than external memory to cut bandwidth - removing the exact bottleneck we've been discussing throughout the blog.

OpenAI claims up to 750 tok/s on inference speed - roughly 14x faster than their standard. But my prediction is that it's going to be even more expensive than Fast mode. That is because of limited capacity, lower batch sizing (B=1 or 2) and a much more expensive silicon.

Conclusion

What we see about token pricing isn't arbitrary. The decision basically comes down to two numbers: GPU rent and token throughput, and almost every pricing we see is about pushing the second number up.

Knowing these techniques helps us understand where exactly our money is going and why tokens are priced the way they are.

I haven't even discussed inference optimisation techniques like Speculative Decoding, Quantisation, and variance in model architectures like Dense or MoE (Mixture of Experts), or different types of attention mechanisms such as Grouped Query Attention.

Though we have barely scratched the surface of inference engineering, what I hope is that you have developed a mental model of how exactly models are priced and what the trade-offs are that help us finally decide an LLM's API pricing.

---

Further reading:

• What you should know before using Coding agents- An introductory guide for those who are new to coding agents like Claude Code or Codex
• The layers of modern AI infrastructure- A brief discussion about how modern AI systems are separated into different layers
• Building Neural Networks from first principles- A two series blog where we build Neural networks from absolute ground-up.