信任区域Q伴随匹配
阅读原文· arxiv.orgQAM将离线策略强化学习转化为无记忆随机最优控制问题,但critic引导的脆弱性导致模型崩溃。TRQAM通过投影对偶下降自适应控制预训练流策略的路径空间KL散度,优化信任区域参数λ,并证明路径空间KL可表示为λ的闭式函数。在50个OGBench任务上,TRQAM在离线RL和离线到在线RL中均超越先前方法,离线RL成功率达68%,超过最强基线(46%)。
Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this issue by reformulating into a memoryless stochastic optimal control (SOC) problem with a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading to model collapse. This paper introduces Trust Region Q-Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL with pretrained flow policies through projected dual descent. Specifically, we optimize the trust-region parameter λ in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of λ. As a result, our method can precisely control the exact deviation from pretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%.