As models become bigger, they also get more capable, but at the cost of increased latency, especially during token generation. Any provider serving these models at a large scale simply cannot afford such huge latencies.

Standard decoding starts being a problem as the demand for AI Inference grows. Some researchers at Google came up with an interesting idea: what if we use a small model to predict some tokens, and the larger model verifies them all in one go?

Adding another model on top of an already huge model seems pretty counterintuitive, but it actually results in a 2-3x inference speedup!

Let's first take a quick look at how standard decoding works to understand its limitations. This will help us understand what we're dealing with.

The Limitations of Standard Decoding

First, we write and send a prompt to the LLM. The model fetches its weights from the HBM, and the prompt is converted into tokens, which the model processes in parallel.

The model performs the attention mechanism on the input tokens and stores the result in a Key-Value cache (or KV cache). This KV cache grows linearly with prompt length. The KV cache is then stored in HBM, along with the weights, ready for the next fetch.

Now the model switches to the decode phase. The model fetches the weights, processes the KV cache and generates the first token. This token is what we see on the screen.

The generated token is then fed back into the model along with the existing KV cache to generate the next token.

Each decode step involves re-fetching the KV cache and model weights from the HBM back into GPU memory to generate the next token. This continues until the model produces a 'stop' token signalling the end of the response.

As you might have guessed, this constant fetching of weights and the cache is what slows down token generation - the process is bounded by memory bandwidth. This means that during token generation specifically, the biggest bottleneck in AI Inference is memory bandwidth, and the GPU cores sit idle while the weights are fetched.

While this gets the job done, our hardware remains underutilised, and we can improve system efficiency, reduce inter-token latency (the time between each generated token), and drastically increase token generation speed.

The Speculation Process

Speculative Decoding is an inference optimisation technique where a smaller "draft" model predicts multiple tokens, and the main "target" model verifies them in parallel to speed up token generation.

As the name suggests, the smaller model guesses the next tokens. All those tokens are fed into the larger model for parallel verification.

As we've already seen, decoding is memory-bound, so we don't fully utilise all available GPU cores. By adding another model, we're essentially adding parallelism back into decoding.

A nice feature about Speculative Decoding is that the target model entirely controls the output, not the draft model. In other words, the output would be the same as standard decoding, but using a draft model, the inference speed is much faster.

Here's a video that shows the difference between Speculative Decoding (left) and Standard Decoding (right). The main model is Qwen 3 14B, and the draft model in spec-dec is Qwen 3 0.6B: both quantised to 4 bits and running on Apple M5 through MLX.

Speculative Decoding happens in 3 main steps. Let's look into them one by one.

1) Token Drafting

Once we send the prompt to the system, the draft model starts generating tokens autoregressively, the same way as standard decoding. The same prompt is also sent to the target model, and both generate their own KV caches.

Since the draft model is physically smaller, fetching the model weights and cache from memory is much easier.

Let's say that the draft model generated $\gamma$ tokens. For each token, the model assigns a probability distribution. The actual draft token is whichever one gets sampled from that distribution.

For each of these $\gamma$ tokens, we get a discrete probability distribution which we refer to as $q$.

2) Parallel Verification

Now we have to feed these tokens into the target model. This ensures our output remains consistent with the target model.

Now we could feed all these $\gamma$ tokens one by one into the target model. But here's a neat trick: we can feed all of these tokens at once into the target model, which appends them to its own KV cache from before.

We do this because the target model processes all tokens at once; we only need to fetch the weights and the KV cache once. All the draft tokens are checked and verified in a single forward pass.

This is where the speedup in our inference really comes from.

After processing the draft tokens, the target model produces its own probability distribution at each position, which we can assign as $p_1, p_2, p_3 ...$ etc., all in a single forward pass.

3) Rejection Sampling

Now comes the hard part: how does the target model verify whether the generated draft tokens are what we want?

This is particularly important because the draft tokens should match what the target model would've generated in standard decoding. This means that we have to accept or reject the tokens generated by the draft model.

Simple Case - Greedy Decoding

The simplest and easiest way to accept or reject tokens is to just check at each corresponding index whether the token matches or not.

Here, the draft tokens are fed into the target model alongside its existing KV cache. But because of the attention mechanism, the target's prediction at each position depends only on its previous tokens. So even if it reads all of the draft tokens at once, it processes them in such a way that a token cannot see its future tokens. The target model computes its own prediction, independent of the draft model. Here, its top prediction is "the", not "a". And since the two don't match, this token is rejected.

'a' was assigned a different probability than 'the'. It's purely working off "fox jumps over" as if it was generating fresh, with absolutely no knowledge of what the draft guessed there.

And since we encounter our first token mismatch, everything after that is rejected, even if the draft token was matching the target token.

Why?

Because the target's prediction "happy" was computed assuming "a" came before it. But since the target generated "the", that original context is no longer valid. Getting the correct prediction for "happy" would require re-running the target with the corrected sequence - which we haven't done yet.

The further tokens are then regenerated in the next round.

Why this isn't used in production

This method seems pretty easy to work with: just compare the tokens and speed up the decoding process until we're done. But this isn't practical and would give us the wrong results.

LLMs don't pick the next token with the highest probability. They sample the next token from the probability distribution to get a natural and varied output. This process is controlled by temperature, top-k and top-p parameters.

Doing so otherwise would result in a deterministic and often repetitive text. We need a better formula that works when the tokens are sampled with certain parameters, which is what the general case handles.

The General Case

We now consider a general case where an LLM samples the token using temperature/top-p/top-k; instead of simply taking the argmax, which is what really happens in a production system.

Instead of simply comparing and eliminating tokens, we use their probabilities assigned by each model and use them to accept or reject tokens.

We already have a set of tokens and probabilities from our previous step. We're going to use a rule to accept or reject tokens as follows:

$$\text{accept if } u \leq \min{(1,\frac{p_i(x_i)}{q_i(x_i)})}, \space \space u \sim \text{Uniform}(0,1)$$

Using this rule and our tokens, we apply this formula on the probabilities:

Notice one thing: a draft token is rejected only when the target model is less confident than the draft model.

If the target model likes a token at least as much as the draft model did, it's accepted outright.

Once again, the target model was less confident about the token "a" than the draft model (14%). In our walkthrough, we're assuming that the draw went against it and it was rejected. And since that token was rejected, subsequent tokens after this, "happy" in this case, are also rejected.

Replacing the rejected token

Once a token is rejected, all of the following tokens, which were dependent on that first token, are rejected as well. We sample one token from the target model instead of the remaining tokens.

I haven't shown the entire probability distribution for the target $p$ and draft $q$ at position 4 (where "a" is).

Let's consider the following probability distribution at position 4:

As we saw previously before, the Draft model was 70% confident the next token should be "a", whereas the Target model was only 10% confident about it. And this is why this token was rejected.

So how do we find out what the next token should be ?

We first compute a new residual distribution $p'$ using the formula below, then sample the replacement token from it:

$$p'(x) = \text{norm}[\max(0, p(x) - q(x))]$$

Applying this formula to our residual probability distribution:

Our total leftover is 0.5 + 0.1 = 0.6. We divide each clipped value this by this total to find out our normalised probabilities.

Notice one thing: the new probability of "a" is 0. This is because the draft model already predicted it earlier, but the target model had a low confidence on it. So it gets a zero chance of replacement, or being sampled again.

Our new tokens which can be replacements are "the" and "many". They were under-proposed before.

$$\begin{gathered} p'(\text{the}) = 0.50/0.60 = 0.833 \\ p'(\text{many}) = 0.10/0.60 = 0.167 \end{gathered}$$

Originally, "the" had a higher probability score assigned by the target model (or the model was more confident in it.) And so even after resampling,"the" seems to have a higher probability of 83% than "many" which has only 17%.

This is the mechanics that guarantees losslessness.

Had we removed the draft model entirely and only used the target model as standard decoding, we would be sampling the next token $x$ from the probability $p(x)$.

With Speculative Decoding, that same token can now arrive in two ways. Either it is accepted directly from the draft model, or rejected-then-resampled from $p'$.

The math works out in such a way that the combined probability is exactly $p(x)$ again. This is the actual lossless guarantee.

4) A Bonus Token

We just observed that if a draft token is rejected, the target model resamples a new token from the residual probability distribution. This happens even if none of the draft tokens is accepted.

But in the case where all the $\gamma$ tokens are accepted, our mechanism allows us to generate one more token right from the target model.

How does this happen?

Let's assume that the draft tokens $x_1, x_2, ..., x_{\gamma}$ are all valid draft tokens and the target model accepts all the tokens.

While processing these tokens, the target model has already computed the probabilities $p_1, p_2, ..., p_{\gamma}$. Additionally, the target model has also computed one extra probability distribution .

Since the weights and the KV cache are already prefetched from the HBM, we can sample one more token $x_{\gamma +1}$ for free, resulting in $\gamma+1$ tokens.

So all in all, here's what we finally got from speculative decoding:

Even in the worst case where no draft token is accepted, the mechanism has ensured at least 1 token is generated by the target model. Super cool!!

How Much Faster, Really?

We just observed that speculative decoding can significantly speed up token generation. But how much faster does it generate exactly?

Let's define $\alpha$ as the acceptance rate - the probability that a single draft token is going to be accepted. It depends on how much the draft model's distribution $q$ matches the target model's distribution $p$, and how predictable the text is.

A code-completion task with greedy decoding sees a much higher $\alpha$ than open-ended creative writing.

Speculative Decoding happens in multiple rounds, and it keeps on happening until an end-of-sequence token gets accepted into the output.

The expected length $\tau$ of the generated tokens is defined as the average number of tokens we can get each round, over many rounds, since the number of generated tokens can vary between different rounds. Its formula is as follows:

$$E[\tau] = \frac{1 - \alpha^{\gamma + 1}}{1- \alpha}$$

Now, the speedup side. While standard decoding needs one target forward pass per token, speculative decoding spends $\gamma$ cheap draft forward passes plus one target forward pass per round.

If we take $c$ as how cheap the draft model is to run compared to the target model (say the draft is 10x smaller, so $c \approx 0.1$), the speed up is computed as:

$$\text{Speedup} \approx \frac{\text{E}[\tau]}{\gamma c + 1}$$

Let's try some values. With $\gamma = 5$, $c = 0.1$, and a draft model that's right about 70% of the time ($\alpha = 0.7$):

$$E[\tau] = \frac{1 - 0.7^6}{1 - 0.7} \approx 2.94, \quad \text{Speedup} \approx \frac{2.94}{1.5} \approx 1.96\text{x}$$

Increasing the $\alpha$ to 0.85 - a well-matched draft/target pair on a predictable task:

$$E[\tau] = \frac{1 - 0.85^6}{1 - 0.85} \approx 4.2, \quad \text{Speedup} \approx \frac{4.2}{1.5} \approx 2.8\text{x}$$

This is where we get our 2-3x inference speedups. The gap between 1.96x and 2.8x is exactly why acceptance rate matters so much in production.

A draft model well matched with the target model (Qwen 3 14B as the target and Qwen 3 0.6B as the draft), or a more predictable task such as code completion, pushes $\alpha$ up, and the speedup climbs with it.

Applications

Almost every company/frontier lab running inference at large scale is using Speculative Decoding: OpenAI, Anthropic, Alibaba, DeepSeek, Modal, Baseten and many more. If you're serving your LLM on a large scale, this technique is a strong candidate out of many ways to optimise your inference.

Furthermore, almost all inference engines support Speculative Decoding: vLLM, SGLang, TensorRT-LLM. Speculative Decoding is one of the most reliable ways to reduce latency without sacrificing output quality.

Here are a few of the strongest use cases of Speculative Decoding:

• AI Agents: Agents make multiple LLM calls, access tools, and feed skills and prompts to the model. The latency between each of these steps compounds. With Speculative Decoding, each LLM call can be 2-3x faster, resulting in a faster agentic workflow.
• Code Generation: Code contains highly predictable patterns. The syntax stays largely consistent (all variables are defined in the same way, the logic for loops and conditionals stays the same, etc.) If the Draft model's acceptance rate is high, we can get effectively more tokens in every single pass, resulting in much faster Code Generation.
• Cost Efficiency: With this technique, if you're an inference provider, you are essentially able to better utilise your hardware, effectively squeezing more tokens out of it. If the same hardware can serve 3x more inference requests in a time window, that reduces the overall cost to run the process and serve those requests. This directly translates to lower inference API costs. I've discussed this in more detail in this blog.

Trade-offs

Like all nice things in this world, Speculative Decoding comes with a cost. Let's look at what its trade-offs are.

1. Draft Model Compatibility: The tokeniser of both the target and the draft model has to be the same. A fundamental mismatch in their tokenisation process would result in a much lower acceptance rate, which removes the whole point of Speculative Decoding.

   This also means that the Draft Model has to be tuned well to achieve the right balance between speed and output quality. An overly simplistic draft model can result in poor token verifications.

2. Engineering overhead: As you might have already guessed, adding a second model and managing it introduces significant engineering complexity. You have to synchronise the two models, manage memory usage and handle errors efficiently, especially when serving in large-scale inference systems.

3. High Batch Sizes: Speculative Decoding has been shown to work poorly if you are serving a large number of batches or requests. At high batch sizes, the GPU is already busy with continuous batching. So the extra compute that you would usually get in smaller batch sizes disappears, resulting in us going back to memory-bound inference.

For large-scale inferencing systems, honestly these trade-offs are worth it.

Wrapping up

What we covered in this blog is the ''vanilla" speculative decoding, where we have a separate draft model. Engineers have even come up with specific types of Speculative Decoding such as EAGLE or Medusa.

Newer models, such as DeepSeek v4, have draft layers built within the model itself, called Multi-Token Prediction. This is where the draft layers produce multiple tokens, and the model itself verifies those tokens, giving us back the inference speedup.

The core idea remains the same: guess and generate some tokens, verify those tokens in parallel, and maybe generate one of our own.

A more concrete version of this discussion can be found in the original research paper: Fast Inference from Transformers via Speculative Decoding.

We walked through all the steps of how it happens: token drafting, parallel verification, rejection sampling (in both the simple and the general case), generating a bonus token, and the actual math behind it all.

As the demand for AI Inferencing grows, this is one of the rate optimisation techniques for a real speedup - with almost no quality in outputs.

Thank you for reading!