Speculative Decoding Pipelines: Draft-and-Verify for Production LLMs

The Bottleneck of Autoregressive Generation

Running a large language model (LLM) in production feels like watching a tortoise run a marathon. You feed it a prompt, and then you wait. Token by token. This sequential nature is the defining characteristic of autoregressive generation, which is the standard method where models predict one word at a time based on all previous words. For every single token output, the entire massive neural network must perform a forward pass. This process is not compute-bound; it is memory-bound. The GPU spends more time waiting for data to move from high-bandwidth memory (HBM) to the processing cores than it does calculating.

If you are serving requests at scale, this latency adds up fast. A user typing a query expects an instant response, not a slow drip-feed of text that takes seconds per word. Traditional optimization tricks like quantization help, but they hit diminishing returns. To break through this wall without sacrificing the quality of the larger, smarter models, engineers turned to an old concept from computer architecture: speculative execution. In CPUs, this means guessing what code will run next and preparing it early. If the guess is wrong, you discard the work. If it’s right, you save cycles. Speculative decoding applies this exact logic to LLM inference pipelines to accelerate token generation using a draft-and-verify mechanism.

How Draft-and-Verify Works

The core idea behind speculative decoding is simple but powerful: use a smaller, faster model to guess the next few tokens, and then let the large, accurate model verify them in parallel. Instead of generating one token at a time, the system generates a batch of candidate tokens and checks them all at once.

The pipeline operates in three distinct phases:

1. Draft Generation: A small "draft" model (often called a speculator) looks at the current context and predicts the next K tokens. These might be 3 to 10 tokens. Because the draft model is much smaller-think Gemma2-2B-it instead of a 9B or 70B parameter model-it runs significantly faster. It doesn't need to be perfect; it just needs to be fast.
2. Parallel Verification: Here is the magic trick. The large target model receives the original context plus the entire sequence of draft tokens as input. In a standard transformer, a single forward pass produces probability distributions for the next token at every position in the sequence. So, instead of running K separate passes to check K tokens, the target model runs one pass and outputs probabilities for all K positions simultaneously.
3. Rejection Sampling: The system compares the probabilities. For each draft token, it checks if the target model agrees with the draft model's choice. If the target model assigns a higher or equal probability to the draft token than the draft model did, the token is accepted. If the target model disagrees (assigning a lower probability), the token is rejected with a specific probability ratio. Crucially, if any token is rejected, that token and all subsequent draft tokens are discarded. The target model then generates its own correct token for that position, and the cycle restarts.

This process guarantees that the final output distribution matches exactly what the large model would have produced on its own. You get the speed of the small model with the quality of the large model.

The Math Behind Acceptance Rates

Understanding why this works requires looking at the rejection sampling algorithm. Let’s say the draft model proposes the sequence "discovered a breakthrough."

• Token 1: "discovered". Draft probability ($P_{draft}$) = 0.6. Target probability ($P_{target}$) = 0.8. Since $0.8 \ge 0.6$, the target is more confident. Accept.
• Token 2: "a". $P_{draft}$ = 0.7. $P_{target}$ = 0.75. Since $0.75 \ge 0.7$, accept.
• Token 3: "breakthrough". $P_{draft}$ = 0.5. $P_{target}$ = 0.2. The target thinks this is unlikely. Since $0.2 < 0.5$, we reject this token with probability $1 - (0.2/0.5) = 0.6$. If rejected, we discard "breakthrough" and everything after it. The target model then generates its preferred token, say "new," and the loop continues from there.

In the best-case scenario, the target model accepts all K draft tokens. In that case, you generate K+1 tokens (K drafts + 1 new token generated by the target during verification) with only one forward pass of the large model. With K=5, that is a 6x speedup over standard autoregressive generation. In real-world production systems, acceptance rates vary, but speedups of 2x to 3x are common depending on how well the draft model aligns with the target.

Architectural Innovations: Medusa and Beyond

Traditional speculative decoding relies on two separate models: a draft model and a target model. This doubles the memory footprint because you have to load both sets of weights into VRAM. To solve this, researchers introduced architectures like Medusa, which is an architecture that adds multiple prediction heads directly onto the base LLM to enable parallel token speculation.

Instead of loading a separate small model, Medusa attaches lightweight feed-forward layers (prediction heads) to the last hidden layer of the main model. When the model processes the context, these heads simultaneously predict multiple future tokens. This creates a tree of possible continuations in a single inference pass. Because the heads share the backbone of the main model, there is no extra memory cost for loading a second model. This makes Medusa particularly attractive for constrained environments where VRAM is scarce.

Other innovations include MagicDec and adaptive Sequoia trees, which optimize speculative decoding for high-throughput regimes by dynamically adjusting the speculation depth based on confidence levels. These methods aim to maximize the number of accepted tokens while minimizing the computational overhead of verification.

Implementing Speculative Decoding in Production

You don’t need to build this from scratch. Modern inference frameworks have baked speculative decoding into their core engines. If you are using vLLM, a popular high-throughput serving engine, it supports speculative decoding specifically optimized for reducing inter-token latency in medium-to-low QPS workloads. vLLM handles the complex scheduling and memory management required to swap between draft and target contexts efficiently.

The Hugging Face Transformers library refers to this technique as assisted generation, allowing developers to easily implement speculative decoding with minimal code changes. You simply specify the assistant model (the draft model) alongside the target model. For example, pairing Gemma2-2B-it as the assistant with Gemma2-9B-it as the target can yield significant throughput improvements on 8xA100 GPU clusters.

When implementing this, consider these practical tips:

• Model Similarity Matters: The draft model should be from the same family as the target model. Using a Llama-based draft for a Mistral target often results in low acceptance rates because their internal representations differ too much.
• Sequence Length: Speculative decoding shines in long-context scenarios. Short prompts don't benefit as much because the overhead of switching contexts outweighs the gains.
• Hardware Constraints: Ensure your GPU has enough bandwidth to handle the increased memory traffic. While speculative decoding reduces compute steps, it increases memory reads/writes due to the parallel verification step.

Why This Matters for Edge and Mobile AI

As AI moves from the cloud to the edge, speculative decoding becomes even more critical. On-device models must be small to fit in mobile RAM, but users still expect intelligent responses. By using a tiny local model as a drafter and occasionally verifying with a slightly larger local model-or even a remote server-you can achieve near-instant responsiveness.

Applications like real-time translation, interactive chatbots, and gaming NPCs benefit immensely from reduced latency. Users perceive a 200ms delay as noticeable lag. Speculative decoding can cut that latency in half, making interactions feel fluid and natural. Furthermore, since the draft model is smaller, it consumes less power, extending battery life on mobile devices.

Troubleshooting Low Acceptance Rates

If you implement speculative decoding and see little improvement, the issue is likely low acceptance rates. Here is how to diagnose and fix it:

• Check Temperature Settings: High temperature values make the target model more stochastic, leading to more rejections. Lower temperatures generally increase acceptance rates.
• Verify Model Alignment: Ensure the draft model is fine-tuned on similar data as the target. Mismatched training distributions lead to divergent predictions.
• Adjust Draft Length (K): If K is too large, the probability of hitting a rejection early in the sequence increases. Start with K=3 or K=4 and tune upwards.