监控内部独白:探测轨迹揭示推理动态
阅读原文· arxiv.org本研究针对大型推理模型(LRM)思维链监控不可靠的难题,提出“探测轨迹”方法。该方法通过在生成每个令牌时评估探测器,捕捉模型隐藏表征的演变轨迹。研究发现,结合完整轨迹的预测比基于单一点的静态预测更能准确区分模型未来行为。通过提取波动性、趋势等信号处理特征,模型状态区分度得到显著提升。同时,基于模板的训练数据可替代昂贵的动态生成数据,且采用最大池化操作能实现高达95%的AUROC性能并形成稳定轨迹。在安全与数学等四个数据集上的验证表明,该轨迹特征能编码任务动态,为监控LRM行为提供了有效补充框架。
Large Reasoning Models (LRMs) introduce new opportunities for safety monitoring through their Chain of Thought (CoT) reasoning. However, CoT is not always faithful to the model's final output, undermining its reliability as a monitoring tool. To address this, we investigate the hidden representations of LRMs to determine whether future behavior can be predicted from prompt and CoT representations. By evaluating a probe at each generated token, we construct a probe trajectory, the continuous evolution of a concept's probability across the reasoning process. We find that future model behavior is more distinguishable when examined over the full trajectory than from a single static prediction. To characterize these temporal dynamics, we extract signal-processing features that capture volatility, trend, and steady-state behavior, significantly improving the separation of future model states. We also present two methodological insights. First, template-based training data achieves near-parity with dynamically generated model responses, eliminating the need for a costly initial inference and labeling. Second, the choice of pooling operation is critical: average-pooling and last-token methods collapse to near-random performance, while max-pooling achieves up to 95% AUROC and yields stable probe trajectories. Using four datasets and four reasoning models across the domains of safety and mathematics, we demonstrate that trajectory features encode task-specific dynamics that improve outcome separability. These findings establish probe trajectories as a complementary framework for monitoring LRM behavior. Warning: This article contains potentially harmful content.