There’s a failure mode I see constantly in this field: reading a paper and believing you understand it. I used to do it too. You read the EAGLE3 paper, nod along at “feature-level autoregression with a reduced draft vocab,” and feel like you get speculative decoding. You don’t. I certainly didn’t — not until I trained a drafter myself, watched its acceptance plateau at a number the paper never mentions, and spent two days finding out why.
So this is how I actually learn a new technique now: I implement it. When a new inference idea lands, I don’t add it to a reading list — I build it, train it, deploy it, and watch it break in my exact serving regime. This post is the field journal from doing that across the whole speculative-decoding lineage — EAGLE3, MTP, and the block-diffusion frontier (DFlash) — including the unglamorous parts the papers leave out. The newest entry, DFlash v2, shipped four days ago, and I’m bringing it up on the latest SGLang right now.
Table of contents
Open Table of contents
Why speculative decoding is the lever
Start with the bottleneck, because it’s the whole reason this family of techniques exists. At batch size 1, autoregressive decode is memory-bandwidth-bound: to emit one token you stream every weight of the model out of HBM. For a 72B in FP8 that’s ~72 GB per token; on an H100’s ~3.35 TB/s that’s a hard floor near 21 ms/token before any math. The tensor cores sit ~95% idle. You bought a teraflop monster and it’s doing a memory copy.
Speculative decoding fixes the arithmetic intensity. A small draft proposes the next K tokens; the big target verifies all K in one forward pass (the extra positions are nearly free compute on a memory-bound pass); a verification step keeps the longest correct prefix and corrects the first mismatch. At greedy/temperature-0 it’s bit-exact — identical output, just faster. That losslessness is why it became the default acceleration layer in vLLM, SGLang, and TensorRT-LLM rather than a niche trick.
And the single number that defines the win is the mean acceptance length τ — how many tokens you get per target forward. τ = 2 means ~2× decode. Everything else — kernels, all-reduce, graph capture — is a rounding error next to τ. That reframing is the most important thing I internalized, and I only really believed it after measuring it.
The lineage, by building each one
I worked through the family the same way each time: read it, then train it against a 72B target I profile against, then measure τ honestly.
- Medusa — bolt N independent heads onto the final hidden state. Cheap, but the heads don’t see each other, so the joint draft is incoherent and acceptance is mediocre. The “cheap fallback” tier.
- EAGLE / EAGLE3 — draft in feature space, not token space: a tiny transformer head autoregressively predicts the target’s next hidden state and reuses the target’s LM head. EAGLE3 adds a direct token-prediction objective and fuses low/mid/high target layers into the draft’s conditioning. This is the battle-tested baseline I trained first — a ~1-layer head, ~1.4B params, with a reduced draft vocab (the draft only has to be good at common tokens; the verify catches the rest).
- MTP (DeepSeek-V3) — stop bolting a draft on; train the model to predict multiple tokens during pretraining, with sequential causal modules that keep the chain. The draft is the model, perfectly aligned, no distribution mismatch by construction. This is where the field is converging.
- Diffusion (DiffusionGemma, June 2026) — drop left-to-right causality and denoise a whole block in parallel. ~4× faster than its AR sibling, but still trails it on reasoning benchmarks. The high-variance bet.
- DFlash — the synthesis I find most interesting, and the one I’m on now. More below.
What implementing actually taught me (and reading didn’t)
Here is the part you cannot get from a paper. Every one of these cost me real GPU-hours.
Data beats architecture, by a lot. Everyone ships the same EAGLE3 head format, and the field’s heads were all stuck at acceptance ≈ 1.6. I assumed for too long the gap was architectural. It wasn’t. An EAGLE3 head learns to predict the target’s own output distribution — so if you train it on generic “continue this passage” prose but serve long analytical answers, it learns the wrong distribution and plateaus no matter how long you run. The fix is self-distillation on the real distribution: run the target on prompts that match what you actually serve, and train the draft on the target’s own greedy outputs. That single change moved acceptance more than every hyperparameter combined.
The max-length truncation trap silently trains on nothing. Your supervision lives at the end of each sequence. If max_length is shorter than prompt + answer, the answer gets truncated, the loss mask empties, and the dataset filter drops the sequence — so a long-prompt dataset can filter to zero while training “runs” with a plausible-looking loss and acceptance ≈ 0. I burned a run on this. The lesson: tokenize a sample before you launch, and set max_length ≥ p95(prompt + answer). Never treat it as a memory dial.
OOM doesn’t crash multi-GPU training — it hangs it. One rank OOMs, the others spin forever on the NCCL collective: 100% util, low power draw, asymmetric memory, no new logs, and the distributed timeout often never fires. The tell is power draw, not utilization. The fix is gradient checkpointing, never shrinking the sequence (that triggers the trap above).
The version-drift tax is real and sequential. Pointing a research trainer (SpecForge) at a fast-moving engine (SGLang) is a chain of breakages, each surfacing only after you fix the last — renamed modules, a ServerArgs kwarg the backend still emits, a return tuple that grew from 3 to 4 elements, a request field replaced, the new overlap scheduler deferring input_ids materialization. Seven, in my case, to move one trainer across a few minor SGLang versions. It’s terrain, not misfortune; capture the fixes as a patch so you only walk it once.
Resuming an annealed schedule learns nothing. A naïve --resume reloads the annealed LR (cosine decayed to ≈ 0), so the warm-start relearns nothing. Load the weights but build a fresh warmup+cosine schedule — and verify the warm-start by the initial loss (a correct resume picks up at the trained level, not random-init). Then don’t panic at the warm-up dip: re-warming to full LR deliberately kicks the weights out of their old optimum, so accuracy drops before it breaks past the previous peak. Kill the run at the dip and you’ve killed one that was about to surpass itself.
Almost every spec-decode number is a lie outside its exact regime. This is the most important and least-discussed lesson. A head/recipe that gained +22% at TP=4 went to +5.9% at TP=8 — worse than no spec at all, because the 8-way all-reduce barrier in the verify ate the gain. A +41% “win” on H100 was a mirage — the real regime was H200, where the same image went net-negative. The optimal draft depth even flipped between TP degrees. Validate only in the exact production regime — same GPU, same TP, same quantization. A cheaper benchmark will cheerfully tell you you’ve won when you’ve lost.
Correctness gates punish numeric drift. fp8 weight divergence from a bf16 reference, and torch.compile on the spec path, each silently broke greedy-exactness for me. Match the reference numerics: bf16, no compile on the spec path.
DFlash: the frontier I’m on now
EAGLE3 drafts one token at a time — K head-forwards for K tokens — which is why EAGLE heads are kept shallow. DFlash (Z Lab’s block-diffusion speculative decoding) proposes a whole block of K tokens in a single forward pass: it takes a [token, MASK, MASK, …] template and denoises the block at once. The key idea is KV injection — the target’s per-layer hidden states are projected and prepended to the draft’s K/V at every layer, so the small drafter skips re-modeling the context and focuses purely on the next block, conditioned on the target throughout (not just at the input, the way EAGLE3 fades).
I trained a DFlash drafter for a 72B target from scratch — no off-the-shelf one exists, because KV injection ties a drafter to its target’s hidden states. First pass: accept 1.49, undertrained (loss still falling). A warm-start continue (with the fresh-LR fix above) took it to 1.71 — a real +14%, though still short of a mature EAGLE3 field head (~2–2.5). The tell was the plateau: training accuracy flattened while loss kept making small new lows, which means I hit the ceiling of my training set, not the epochs. The lever from here is more and better-matched data — exactly the §“data beats architecture” lesson, a second time.
Then, four days ago, DFlash v2 landed — Z Lab, Modal, and the SGLang team, shipping DFlash on SGLang’s new Spec V2 engine. Two things make it matter:
- The block-diffusion + KV-injection combination wins on both axes. On Qwen3-4B, a 5-layer DFlash hits roughly the same acceptance as a 5-layer EAGLE3 (~4.0–4.2) but delivers ~3.2–3.3× end-to-end vs EAGLE3’s ~2.1–2.2×, because the draft is one parallel forward instead of K. Ablations show each piece pulling its weight: diffusion-only is faster-but-lower-acceptance, injection-only is higher-acceptance, together they compound.
- Spec V2’s overlap scheduler kills host overhead. The win isn’t only the GPU — it’s overlapping host-side cleanup and KV allocation with GPU work so host↔device synchronization stops stalling the pipeline. The DFlash KV projection is run ahead of the draft forward (“immediate materialization”) with a fused Triton kernel for the norm+RoPE post-processing.
So my next build is exactly that: bring DFlash v2 up on the latest SGLang Spec V2 engine, against my 72B, and measure τ and end-to-end in my real regime — not trust the headline number. That’s the whole ethos of this blog and my roadmap: the efficiency layer between the algorithm and the kernel is where the wins are, and you only understand it by building it.
The takeaways, distilled
- Acceptance length is the speed. Optimize τ; almost everything else is a rounding error.
- A drafter is only as good as its data matches what you serve. Self-distill from the target on your real distribution. This is the lever, twice over (EAGLE3 and DFlash).
- The silent failures cost the most — empty-loss truncation, OOM-hangs, stale preprocessing caches, annealed-LR resumes. None of them error. All of them waste a GPU-day.
- Validate in the exact production regime. TP degree, GPU, and quantization change the answer non-linearly.
- Reading is necessary and insufficient. The paper gives you the idea; the implementation gives you the truth.
The one-token-at-a-time era is ending — not because anyone proved autoregression wrong, but because the GPU was bored the whole time, waiting on memory, and the field finally found honest ways to keep it fed. I’d rather be one of the people building the thing that feeds it than one of the people who read about it.