LongTraceRL:基于搜索智能体轨迹与评分奖励的长上下文推理学习
阅读原文· arxiv.org针对大语言模型在长上下文推理中难以定位关键信息的问题,现有强化学习方法受限于干扰物挑战性不足和奖励信号稀疏。LongTraceRL方法通过知识图谱随机游走生成多跳问题,并利用搜索智能体的轨迹构建分级干扰文档,从而生成更具挑战性的训练数据。其创新性地提出评分奖励,使用推理链上的金标准实体作为细粒度的过程监督,且仅应用于最终答案正确的响应,以此区分推理质量并防止奖励作弊。实验表明,该方法在多个长上下文基准测试中优于强基线。
Long-context reasoning remains a central challenge for large language models, which often fail to locate and integrate key information in extensive distracting content. Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task, yet existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals that cannot supervise intermediate reasoning steps. To address these issues, we introduce LongTraceRL. For data construction, we generate multi-hop questions via knowledge graph random walks and leverage search agent trajectories to build tiered distractors: documents the agent read but did not cite (high confusability) and documents that appeared in search results but were never opened (low confusability), producing training contexts that are far more challenging than those built by random sampling or one-shot search. For reward design, we propose a rubric reward that uses the gold entities along each reasoning chain as fine-grained, entity-level process supervision. This rubric reward is applied only to responses with correct final answers (positive-only strategy), distinguishing the reasoning quality among correct responses and preventing reward hacking. Experiments on three reasoning LLMs (4B--30B) across five long-context benchmarks demonstrate that LongTraceRL consistently outperforms strong baselines and encourages comprehensive, evidence-grounded reasoning. Codes, datasets and models are available at https://github.com/THU-KEG/LongTraceRL{https://github.com/THU-KEG/LongTraceRL}.