同一智能体不同答案:检索增强问答中语料库引发的答案更替审计

HuggingFace Daily Papers(社区热门论文)·2026-08-24 08:00·1天前
AI 导读

检索增强问答系统在索引扩展后可能返回不同答案,即便模型标识、提示词、检索策略等设置完全固定。研究提出快照兼容性审计,通过从跨快照分歧中减去同快照重复分歧来估算超额答案更替。在400题Natural Questions预注册研究中,归一化精确和盲语义超额更替分别为6.44和10.25个百分点,而精确匹配准确率仅变化-1.50个百分点。

HuggingFace Daily Papers(社区热门论文)
68AI 编辑部评分,满分 100

同一智能体不同答案:检索增强问答中语料库引发的答案更替审计

2026-08-24 08:00· 1天前
AI 导读

检索增强问答系统在索引扩展后可能返回不同答案,即便模型标识、提示词、检索策略等设置完全固定。研究提出快照兼容性审计,通过从跨快照分歧中减去同快照重复分歧来估算超额答案更替。在400题Natural Questions预注册研究中,归一化精确和盲语义超额更替分别为6.44和10.25个百分点,而精确匹配准确率仅变化-1.50个百分点。

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the Snapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only -1.50 points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org

arXiv检索增强搜索论文/研究