Paths diverge.
Thought goes on.

Speculative decoding lets a large language model verify multiple candidate tokens in one pass, reducing the sequential cost of token-by-token generation. With parallel drafters such as DFlash and DSpark, an entire candidate block can already be proposed in one forward pass. The next question is how to train these drafters so that more of their proposals are accepted.

On-policy distillation (OPD) offers a useful path: let a drafter learn on histories generated by its own policy. Conventional pipelines, however, still collect complete responses through successive draft-and-verify rounds before updating the drafter. The sequential dependencies of inference enter the training loop.

We introduce SpecOPD, which collects fresh on-policy supervision across independent branches while reusing their shared-prefix computation. In the paper's evaluated settings, it delivers three different kinds of gains:

• 16.09× supervised-token throughput over sequential-rollout OPD.
• Up to 41.45% higher five-benchmark mean acceptance length over the original drafter.
• Up to 3.56× measured generation throughput over autoregressive decoding with a 27B target.

These numbers measure training capacity, proposal quality, and inference speed, respectively. Their common starting point is the sequential dependency in supervision collection.

Why drafter training still waits

Speculative decoding pairs a lightweight drafter with a target model. The drafter proposes candidates; the target verifies them in parallel and accepts an eligible prefix. The verification rule preserves the target's output distribution, while longer accepted prefixes let each target pass advance generation further.

Parallel drafters reduce proposal cost, but acceptance length still depends on alignment with the target. Training only on fixed target-generated responses exposes a drafter to histories different from those it produces itself. OPD supplies target supervision on draft-generated histories, helping the drafter learn from its own prediction errors.

In a typical rollout-and-distill pipeline, the accepted prefix from one verification determines where the next round begins. A complete response can require many dependent draft-and-verify rounds before training and weight updates follow. This schedule limits the sustained, larger-batch computation that GPUs handle efficiently.

Enlarge figure

Figure 1 | Measured execution profiles and training throughput. The upper and lower traces on the left show sequential-rollout OPD and SpecOPD on separate time axes. Throughput on the right comes from independent, unprofiled timing runs.

In the illustrated windows, the sequential pipeline alternates rollout, training, and weight updates, while SpecOPD batches drafting and target scoring. Mean recorded GPU activity rises from 26.37% to 77.23%. This leads to a practical question: does on-policy supervision require collecting a complete, verified response first?

A draft block needs only its own history

SpecOPD starts from a simple observation: supervising a draft position requires the context preceding that position.

That context can combine a fixed reference prefix with fresh tokens sampled by the current drafter. The target supplies supervision along this sampled history, without accepting or rejecting proposals to advance a complete rollout.

Consider the reference response 'Enjoy a quiet evening at home'. At an anchor after 'a', the drafter samples 'short evening walk':

Even when a sample diverges from the reference response, the target can provide its next-token distribution for that sampled history. Training can therefore use the drafter's own histories without waiting for a full rollout.

We can also sample another branch, such as 'with soft music', after the reference token 'evening'. Each branch depends on its own reference prefix and sampled tokens; neither needs the other's verification result. Many anchors can thus supply batched branches and target supervision.

On-policy here means fresh continuations sampled from the current drafter at fixed reference contexts. The outer context remains fixed, while the drafter learns on its own branch histories.

Share computation while preserving branch histories

Recomputing every complete reference prefix for every branch would offset part of the parallelism gain. SpecOPD organizes the reference sequence as a shared trunk and attaches sampled continuations at their anchors, forming a proposal tree.

Branches reuse computation for common ancestors while retaining separate histories after divergence.

Enlarge figure

Figure 2 | Sample reference anchors, generate continuations with the current drafter, and attach them to the shared trunk. Future reference tokens are not visible to the sampled branches.

Different target architectures represent history differently and require corresponding execution mechanisms:

The common principle is shared-prefix computation with independent branch histories. For linear attention, indexed state replay avoids executing the entire prefix again for every branch, extending branch-parallel supervision to hybrid targets.

Train for longer accepted prefixes

Efficient execution also needs a useful learning signal.

Speculative decoding accepts a continuous prefix. Once an earlier candidate is rejected, correct predictions at later positions cannot extend that round's accepted prefix. Independent token accuracy therefore does not fully capture the value of a draft block.

SpecOPD uses the E2E-TV acceptance-length surrogate. It measures overlap between draft and target distributions at each position and aggregates prefix products along the sampled branch:

Intuitively, larger prefix products encourage proposals that can be accepted consecutively. Training combines this signal with fresh sampled branches, while actual gains are evaluated directly through serving acceptance length.

Two sources of training efficiency

On Qwen3.8-27B with DFlash V2 and eight high-performance GPUs, the native-configuration comparison improves both update throughput and supervision per update:

Branch parallelism reduces collection waits and supplies more supervised positions per update. Tokens count only predictions contributing to the training loss, excluding context-only tokens.

The comparison includes each method's training workflow under its native configuration and excludes offline reference preparation. The 16.09× result measures supervision-processing capacity, rather than a 16.09× reduction in time to matched model quality.

From acceptance gains to inference speed

We evaluate SpecOPD with 8B full-attention, 27B full–linear, and 125B sparse–linear targets; DFlash, DSpark, and DFlash V2 drafters; and four task domains: dialogue, coding, mathematics, and agentic workloads.

The five benchmarks are MT-Bench, HumanEval, MATH-500, SWE, and Terminal. Completed comparisons show up to 41.45% higher five-benchmark mean acceptance length and up to 81.51% higher agentic-task acceptance length. The latter measures continuation efficiency from recorded contexts.

Gains vary by architecture and task. Five-benchmark mean acceptance-length improvements reach 3.48% on 27B and 5.01% on 125B. A separate study with public DFlash V2 initialization and an FP8 Flash-Next target reaches best observed task–temperature gains of 7.72% at block four and 15.57% at block eight, each relative to its matched initialization.

Enlarge figure

Figure 3 | Measured generation throughput with a 27B target across workloads and concurrency levels, using BF16, TP=1, FA3, and CUDA Graphs.

In this serving study, SpecOPD reaches 3.56× the generation throughput of same-GPU autoregressive decoding. Across the five workloads, geometric-mean speedups are 2.97×, 2.23×, and 1.35× at concurrency one, eight, and 32, respectively.

Compared with the original DFlash V2, the geometric-mean throughput gain reaches 2.83%. These comparisons distinguish the acceleration from speculative decoding itself from the additional benefit of further training the drafter.

Longer acceptance does not always produce a proportional throughput gain. Concurrency, drafting cost, and target verification cost all affect end-to-end performance, so we report measured serving throughput alongside acceptance.

Why on-policy learning matters

We compare SpecOPD with off-policy CE and off-policy E2E-TV trained on fixed target-generated responses.

At matched optimizer steps and reference-sequence batches, SpecOPD improves mean acceptance length by up to 15.71% over off-policy E2E-TV and 6.96% over off-policy CE. The recipes differ in compute allocation and gradient paths; the comparison measures their evaluated training behavior.

Enlarge figure

Figure 4 | Training loss, acceptance length on three benchmarks, and accepted-prefix survival changes. Shading gives pointwise 95% question-cluster bootstrap intervals.

All three runs reduce training loss, yet the off-policy baselines can lose acceptance as training progresses, while SpecOPD improves measured acceptance.

At each method's best evaluated checkpoint by mean acceptance length, SpecOPD improves prefix survival at all seven draft positions. At position seven, survival rises by 4.33 percentage points over initialization. This connects the training gains to how far an accepted prefix can progress and motivates checking actual decoding behavior alongside loss.

From parallel drafting to parallel training

SpecOPD builds on a local observation: a draft block can learn directly from target supervision on its own sampled histories.

Reference anchors supply independent branches; proposal trees expose shared-prefix computation; history-consistent execution supports full, sparse, and linear attention; and an acceptance-length surrogate guides updates. Together, these designs turn sequential full-response collection into training computation that is better suited to GPU batching.

As target architectures evolve, improving drafters efficiently and translating better acceptance into dependable serving gains remain important problems. SpecOPD provides a branch-parallel training path for that process.

Paper

Read the SpecOPD paper

SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding. Figures and experimental values correspond to the paper's public edition. Acceptance length includes bonus tokens.

@misc{liu2026specopd,
  title = {{SpecOPD}: Branch-Parallel On-Policy Distillation
           for Speculative Decoding},
  author = {Liu, Tianyu and Zhang, Jun and Huang, Wenjie and
            Shen, Yuhao and Xu, Tianqi and Hao, Xinyue and
            Gu, Hongyaoxing and Liu, Xuyang and Zhang, Baolin and
            Zhu, Yating and Ge, Shuang and Dai, Jun and
            Chen, Lei and Li, Yue and Wan, Mingchen},
  year = {2026},
  url = {https://github.com/QwenApp/SpecOPD}
}