诊断答案正确的长链式推理训练轨迹中的有害延续
阅读原文· arxiv.org本研究探讨用于大语言模型监督微调的长链式推理轨迹。研究发现,即使推理轨迹答案正确,其在结论后仍继续的推理部分也可能对训练产生有害影响,导致微调结果显著不同。这种现象被定义为“有害延续”,其特征是持续的局部不确定性与减弱的终端方向进展不匹配。通过编辑器删除这些有害延续后,基于CoT的微调结果得到改善。研究进一步提出了Harmful Continuation Cut(HCC),作为近似有害延续边界的轻量级代理方法。
Long chain-of-thought (CoT) traces are widely used as supervision for reasoning-oriented LLM SFT, yet answer-correct traces can still lead to markedly different fine-tuning outcomes. We study post-conclusion continuation in answer-correct long-CoT data: a continuation where the answer appears sufficiently supported, but the trace continues with additional reasoning that remains in the supervised target. To test its training effect, we use a delete-only editor to construct answer-preserving suffix removal and compare CoT-based SFT on the original and processed traces. We observe improved SFT outcomes after removing the editor-identified post-conclusion continuation, suggesting that this continuation is harmful to training in our setting. We therefore refer to this empirically supported phenomenon as harmful continuation. Beyond this intervention, we further characterize the removed post-conclusion continuation through uncertainty and hidden-state progress. We observe persistent local uncertainty together with weakened terminal-directional progress, forming an uncertainty--geometry mismatch. Finally, we instantiate Harmful Continuation Cut (HCC), a lightweight boundary proxy that approximates the editor-identified post-conclusion continuation boundary.