精简草稿,多做检索:投机解码的混合树构造
阅读原文· arxiv.org现有投机解码方法为提高接受率而构建庞大草稿树,导致显存带宽和计算开销激增,反而制约了加速效果。动态剪枝虽能降低延迟,但会损失有效候选,无法达到理想接受率。为此,研究提出了Graft补偿框架,通过“剪枝-再嫁接”的机制,利用剪枝释放的计算预算驱动检索操作,用检索到的高预测性令牌补偿剪枝造成的覆盖损失,从而在近零额外开销下恢复接受长度。该方法无需训练且无损性能,在短上下文、长上下文及大规模模型等多种场景下建立了新的性能前沿。实验表明,它在短上下文任务中最高可实现5.41倍加速,并在大规模Qwen3-235B模型上将平均加速比相比EAGLE-3提升了高达21.8%。研究还初步探索了该方法在DFlash分块草稿范式中的应用潜力。
Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41times speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.