# EfficientRollout：面向RL推理生成的自推测解码框架

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-17 08:00
- AIHOT 分数：48
- AIHOT 链接：https://aihot.virxact.com/items/cmqjfnazc02weslmh6ogc0jdy
- 原文链接：https://arxiv.org/abs/2606.18967

## AI 摘要

强化学习成为大语言模型主流后训练范式，但推理生成中自回归解码的长尾延迟成为瓶颈。传统推测解码依赖固定草稿模型，无法适应RL中不断演化的策略分布。EfficientRollout提出系统感知的自推测解码框架：从目标模型量化出草稿模型，无需额外预训练或在线适应；同步采用系统感知的SD开关策略与接受感知的草稿长度自适应。相比加速的自回归基线，该框架将推理生成延迟降低最高19.6%，端到端延迟降低12.7%，且不损失最终模型质量。

## 正文

Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) offers a natural way to address this bottleneck, as it is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.