性能优化基准是否可靠衡量编码智能体?
阅读原文· arxiv.org一项审计研究对GSO、SWE-Perf和SWE-fficiency三个仓库级性能优化基准进行系统性核查。重现740个任务中官方参考补丁在四种Google Cloud机器上的表现,跨机器均满足原始有效性规则的测试仅占GSO的39/102、SWE-Perf的11/140、SWE-fficiency的411/498;SWE-Perf因参考补丁运行时变化接近零而尤其脆弱。排行榜评分规则导致8个公开提交的28组成对比较中有9组排名不一致,SWE-fficiency最差十项任务权重高达58.5%-82.8%。此外,在450个可重现任务中,至少一个提交已匹配或超越参考补丁的比例达85.3%,超越未优化基线达99.8%,揭示了聚合排名掩盖的真实性能差距。
Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.