Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the G-AP (Gap of Aggregate Performance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose SA-PPG (Stratified Aggregate of Per-question Probability Gaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. RailCap instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.
零差距并非恢复:基准污染的分层逐题概率评估与逐步缓解
AI 导读
论文提出SA-PPG分层逐题概率差距评估法,通过采样估计每题求解概率并与干净模型逐题对比,揭示现有G-AP指标存在缺陷,导致此前污染缓解策略的恢复效果被显著高估。研究还提出RailCap缓解策略,在生成中限制回落到贪心轨迹的下一token为次优选项,在多个受污染模型和基准上取得最低SA-PPG。
HuggingFace Daily Papers(社区热门论文)
50
AI 编辑部评分,满分 100零差距并非恢复:基准污染的分层逐题概率评估与逐步缓解
论文提出SA-PPG分层逐题概率差距评估法,通过采样估计每题求解概率并与干净模型逐题对比,揭示现有G-AP指标存在缺陷,导致此前污染缓解策略的恢复效果被显著高估。研究还提出RailCap缓解策略,在生成中限制回落到贪心轨迹的下一token为次优选项,在多个受污染模型和基准上取得最低SA-PPG。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org