# DistIL：基于分布化DAgger的丰富反馈强化学习方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-03 08:00
- AIHOT 分数：61
- AIHOT 链接：https://aihot.virxact.com/items/cmq5e15cb072islt2fka16ze2
- 原文链接：https://arxiv.org/abs/2606.05152

## AI 摘要

现有RLVR仅用单比特反馈判断答案正误，忽略执行轨迹、工具输出等丰富信息。DistIL通过分布化DAgger算法，使学习器局部访问当前策略下状态的专家分布，优化前向交叉熵目标实现序列级信用分配。理论证明前向交叉熵保证单调策略改进和遗憾界，而反向KL或JS散度的自我蒸馏无法做到。DistIL在科学推理、编程和硬数学问题等领域优于RLVR和自我蒸馏基线，并提升Pass@N。

## 正文

Reasoning models have advanced rapidly, but the dominant reinforcement learning from verifiable rewards (RLVR) recipe remains surprisingly narrow: sample many responses and reward each with a single bit indicating whether the final answer is correct. Yet many settings provide rich feedback, including execution traces, tool outputs, expert corrections, and model self-evaluations. We study how to use such feedback through a distributional variant of the classic imitation learning algorithm DAgger, where the learner has local access to an expert distribution on states visited by the current policy. This yields a simple forward cross-entropy objective that admits a blackbox expert and whose sequence-level gradient {conduct rich credit assignment by propagating} future expert-student disagreement back to earlier decisions. We show that prior RL with self-distillation objectives based on reverse KL or Jensen-Shannon fail to guarantee monotonic policy improvement: even when the expert has higher reward, their updates may increase probability on worse actions. In contrast, we show that forward cross-entropy admits monotonic policy improvement and enjoys guarantees on regret. We further show that our objective optimizes a lower bound on teacher-weighted likelihood of success, leading to improved Pass@N. Empirically, our approach, DistIL, improves over RLVR and RL with self-distillation baselines across a variety of domains: scientific reasoning, coding, and solving hard mathematical problems.