# 信任区域Q伴随匹配

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-26 08:00
- AIHOT 分数：47
- AIHOT 链接：https://aihot.virxact.com/items/cmq17v5wp0db7sltrclw9bbim
- 原文链接：https://arxiv.org/abs/2605.27079

## AI 摘要

QAM将离线策略强化学习转化为无记忆随机最优控制问题，但critic引导的脆弱性导致模型崩溃。TRQAM通过投影对偶下降自适应控制预训练流策略的路径空间KL散度，优化信任区域参数λ，并证明路径空间KL可表示为λ的闭式函数。在50个OGBench任务上，TRQAM在离线RL和离线到在线RL中均超越先前方法，离线RL成功率达68%，超过最强基线（46%）。

## 正文

Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this issue by reformulating into a memoryless stochastic optimal control (SOC) problem with a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading to model collapse. This paper introduces Trust Region Q-Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL with pretrained flow policies through projected dual descent. Specifically, we optimize the trust-region parameter λ in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of λ. As a result, our method can precisely control the exact deviation from pretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%.