利用稀疏自编码器的模型内部信号指导LLM后训练数据工程
阅读原文· arxiv.orgSAERL是一个面向大语言模型强化学习的数据工程框架,利用稀疏自编码器从模型内部提取信号。它建模了数据的多样性、难度和质量三个属性,并分别对应批次混合控制、难度排序和质量过滤等具体工程操作。实验表明,在通义千问(Qwen2.5-Math-1.5B)上,SAERL相比标准GRPO平均准确率提升3.00%,并能以减少20%的训练步数达到目标准确率。该方法在不同模型规模和RL算法上均有一致收益,且SAE能跨模型系列和规模有效迁移,证明了模型内部信号作为后训练数据工程信号源的实用价值。
Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.