LaRA:基于逐层表征分析的强化学习后训练数据污染检测
阅读原文· arxiv.org强化学习后训练能提升大语言模型的推理能力,但也可能引入数据污染,影响模型泛化和评估可靠性。现有基于输出层信号的检测方法对RL模型效果不佳。为此,研究提出了LaRA框架,通过逐层分析模型表征来检测污染。该框架引入三个互补指标,分别度量受控扰动下的扰动敏感度、方向坍缩和局部表征刚性。研究发现,污染会导致模型表征在各层出现渐进式几何偏差。基于此,研究开发了一种聚合各层与各指标表征偏差的检测协议,实验表明其性能优于现有的输出级基线方法。
Reinforcement learning (RL) post-training has shown to improve reasoning in large language models (LLMs). However, there has been little exploration on the problem of data contamination in RL post-training, potentially undermining generalization and evaluation reliability of the training process itself. Existing detection methods primarily rely on output-level signals such as likelihood or entropy, which become unreliable for RL-trained models since RL shapes behavior through trajectory-level rewards rather than token likelihoods. We propose LaRA, a layer-wise representation analysis framework for detecting contamination in RL post-trained LLMs. LaRA introduces three complementary metrics, measuring perturbation sensitivity, directional collapse, and local representation rigidity under controlled perturbations. We find that contamination produces progressive geometric deviations across layers, including amplified perturbation sensitivity, stronger directional collapse, and enhanced local rigidity. Based on our findings, we also develop a contamination detection protocol that aggregates representation-level deviations across layers and metrics. Experiments on RL-trained reasoning models show that our protocol outperforms existing output-level baselines for contamination detection.