Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality. In this work, we present a principled analysis of the distributions induced by lossy verification methods. We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification. We further construct a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs. Our code is available at https://github.com/ZhouYuxuanYX/Fast-HSD.
重新审视投机解码中的有损验证:机制、权衡与失败模式
AI 导读
投机解码通过轻量草稿模型并行提议并由目标模型验证,以加速大语言模型推理。新研究对有损验证方法引发的分布改写进行原理性分析,将其归为截断式与协作式两类,并构建诊断评估框架。结果显示截断式方法因分布失真可能显著劣于真实截断采样基线,协作式验证则需控制草稿概率超调以防止低质量输出。
HuggingFace Daily Papers(社区热门论文)
49
AI 编辑部评分,满分 100重新审视投机解码中的有损验证:机制、权衡与失败模式
投机解码通过轻量草稿模型并行提议并由目标模型验证,以加速大语言模型推理。新研究对有损验证方法引发的分布改写进行原理性分析,将其归为截断式与协作式两类,并构建诊断评估框架。结果显示截断式方法因分布失真可能显著劣于真实截断采样基线,协作式验证则需控制草稿概率超调以防止低质量输出。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org