# PBSD：利用特权贝叶斯自蒸馏实现长程信用分配

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-08 08:00
- AIHOT 分数：63
- AIHOT 链接：https://aihot.virxact.com/items/cmq6ad1x406mosl5imxgkqn35
- 原文链接：https://arxiv.org/abs/2606.09348

## AI 摘要

PBSD提出一种贝叶斯校准的自蒸馏方法，用于在稀疏最终奖励下进行细粒度信用分配。它通过验证答案的后验与先验概率比衡量轨迹质量，并利用贝叶斯规则将难以估计的答案侧比率转化为标准学生模型与特权、以答案为条件的教师模型之间的似然比。对该贝叶斯证据分数进行自回归分解，产生每步信号，识别中间推理步骤是支持还是削弱已验证结果。PBSD将稀疏结果监督转化为贝叶斯校准的逐步信用信号，与标准策略优化兼容。实验表明，该方法在领域内和领域外设置中一致提升性能，并有效将知识从短上下文训练迁移到长上下文推理。

## 正文

Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning: trajectory-level rewards verify final correctness but provide limited guidance on which intermediate reasoning steps or tool interactions contribute to the outcome. The difficulty is especially pronounced in multi-turn search agents, where successful trajectories may contain misleading actions and failed trajectories may contain valuable evidence-gathering steps. We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards. PBSD measures trajectory quality through the posterior-to-prior probability ratio of the verified answer and applies Bayes' rule to convert this hard-to-estimate answer-side ratio into a tractable likelihood ratio between a standard student model and a privileged answer-conditioned teacher model. Autoregressive decomposition of this Bayesian evidence score yields turn-level signals that identify whether each intermediate turn supports or undermines the verified outcome. Consequently, PBSD provides a principled and elegant reweighting scheme that transforms sparse outcome supervision into Bayes-calibrated turn-level credit signals, while remaining fully compatible with standard policy optimization. Experiments demonstrate that PBSD consistently enhances performance across both in-domain and out-of-domain settings, and effectively transfers knowledge from short-context training to long-context inference, suggesting that its fine-grained credit assignment mechanism facilitates more effective policy learning and yields improved generalization.