# DeepMind Kaggle 大奖颁给一篇"敷衍"AI论文，引发对基准测试质量的反思 - 来源：Hacker News 热门（buzzing.cc 中文翻译） - 作者：twerkmeister - 发布时间：2026-07-17 20:25 - AIHOT 分数：34 - AIHOT 链接：https://aihot.virxact.com/items/cmrox8uey080nbitoksceoc3a - 原文链接：https://www.kaggle.com/competitions/kaggle-measuring-agi/discussion/724918 ## AI 摘要 一篇被指“敷衍了事”的 AI 论文意外赢得 DeepMind 主办的 Kaggle 大奖，奖金 2.5 万美元。该事件暴露了当前前沿模型评估基准的缺陷——过度依赖召回率（recall），未能真正衡量模型的推理、行动与判断能力。获奖论文呼吁设计更高质量的基准，推动评估体系从单一指标向多维度能力验证转型。 ## 正文 Measuring Progress Toward AGI - Cognitive Abilities Design high-quality benchmarks that go beyond recall to evaluate how frontier models truly reason, act, and judge. Nicholas Kang · Posted 6 days ago Announcing the Measuring Progress Toward AGI: Cognitive Abilities Hackathon Winners Hi everyone, First, thanks for all of your patience as our panel of judges carefully reviewed and discussed all of the submissions to select the winners. Second, THANK YOU to all of the participants in this hackathon! With over 1,000 teams submitting benchmarks across five cognitive tracks, the depth and creativity of this community's work has been truly remarkable. Designing benchmarks that go beyond recall to evaluate how frontier models truly reason, act, and judge is no small feat, and the quality of submissions made judging incredibly difficult. I highly recommend checking out the writeups tab if you haven't already. And finally, I'm very pleased to announce the winners of the Measuring Progress Toward AGI hackathon across the grand prizes and all five cognitive track prizes. Grand Prizes ($25,000 each) 1) MEDLEY-BENCH: Behavioral Metacognition Under Social Pressure by Farhad Abtahi, Abdolamir Karbalaie, Eduardo Illueca-Fernandez, and Fernando Seoane Most AI benchmarks ask whether a model produced the correct answer. MEDLEY-BENCH asks a harder question: does the model know when it might be wrong, and does it update its beliefs for the right reasons? By introducing social pressure into metacognitive evaluation, this benchmark exposes whether models can identify their own uncertainty, maintain correct beliefs when pressured to change them, and revise incorrect beliefs when presented with valid counter-evidence. Congratulations! 2) LearningBench: Measuring Inference-Time Learning in LLMs by Karandeep Singh Existing benchmarks measure what models already know. LearningBench measures how they learn, from scratch, inside a single conversation, on systems that have never existed before. Inspired by ARC-AGI's philosophy that learning is best tested in novel interactive environments rather than static datasets, LearningBench presents models with entirely new text-based systems they must interact with and learn the rules of, measuring in-context learning ability rather than pre-trained knowledge recall. Congratulations, Karandeep! 3) GAUGE: Measuring the Gap Between What LLMs Know and What They Do About It by Arjun Thilak and Ramkumar Muniandi Vallimayil A model that reports 95% confidence and never abstains is more dangerous than one that gets questions wrong, because it gives the human operator no signal that something might be off. GAUGE splits metacognition into monitoring and control using a three-turn "metacognitive staircase" where models predict problem difficulty, solve with a confidence score, then choose to submit or abstain under game-theoretic payoffs. The results are striking: one frontier model achieved the best accuracy and calibration but never abstained once across 270 items, revealing that monitoring without control is a distinct and measurable failure mode. A thoughtful and well-grounded benchmark from the team. Congratulations, Arjun and Ramkumar! Metaproteus evaluates something fundamentally different from most calibration benchmarks: metacognitive knowledge of a model's own output distribution, what a system knows about its own sampling tendencies and behavior, as distinct from what it knows about world facts. By eliciting word-association outputs and then asking a fresh model instance to predict its own responses, Metaproteus reveals distinct failure modes: some models systematically under-claim their own confident outputs, while others over-endorse everything including words they never produce. A creative and important contribution to understanding self-knowledge in AI systems. Congratulations, Joseph! Track Prizes ($10,000 each) Executive Functions 5) Turn Bench by Simon Zhang and Bill Cai Turn Bench uses a collection of turn-based games with intentionally trivial mechanics where the only differentiator is executive function: planning, execution, working memory, inhibition, and adaptive flexibility. Each game has variants that shift diagnostic pressure onto different subcomponents, and variant pairs with opposing demands reveal whether models demonstrate genuine cognitive flexibility or just behavioral defaults that happen to fit one context. The central insight is that executive function is a context-dependent profile, not a monolithic capability that scales with model size. Congratulations, Simon and Bill! LLMs are increasingly deployed in security-sensitive workflows, from triaging incidents to handling credentials, yet existing benchmarks evaluate reasoning in benign settings. SecureExec-Bench systematically probes whether models possess the executive control required to resist adversarial manipulation in high-stakes environments, measuring the gap between general reasoning ability and the capacity to maintain security protocols under pressure. An important benchmark at the intersection of security and cognition. Congratulations, Muhammad! Learning Drawing on the artificial grammar learning paradigm from cognitive psychology, GrammarGym uses synthetic rules to study pattern acquisition independently of meaning. Because these grammars are novel by construction, they provide a controlled way to evaluate learning without relying on memorized knowledge, offering an unlimited source of fake "theories" for LLMs to learn. A clever and principled approach to measuring true learning ability. Congratulations, Raphaël! Most coding benchmarks can't tell whether a model genuinely understands an algorithm or simply pattern-matches against solutions memorized from training data. EphLangBench settles the question by generating unique, procedurally created programming languages per session, with randomized keywords, operators, and notation styles (infix, prefix, postfix), then requiring models to solve algorithmic problems in these never-before-seen languages using only an in-context specification. Since these ephemeral languages cannot exist in any training corpus, success demands real in-context learning: reading a novel grammar, abandoning familiar Python habits, and correcting mistakes from compiler feedback across multiple turns. Across 10 models and 200 problems, the benchmark reveals a dramatic capability spread (7%–89% pass rate) far wider than traditional benchmarks like HumanEval, confirming that ephemeral syntax acts as a powerful multiplier on task difficulty. Congratulations! Metacognition 9) ESFP Benchmark: Epistemic Stance Flexibility Probing in Large Language Models by Binwen Liu and Yilin Ren When you ask a language model "what do experts think about X?" and then "what do you think about X?", you would expect structurally different responses, not just different words, but a different epistemic register. ESFP probes whether models can genuinely shift between reporting and reasoning stances, testing a subtle but crucial dimension of metacognitive flexibility. The results reveal how instruction-tuning pushes models toward being tools that suppress autonomous self-expression, with models varying dramatically in their ability to switch between "tool" and "subject" modes. Congratulations, Binwen and Yilin! A competent assessor working from limited notes does three things at once: forms hypotheses, tracks which inferences are supported by evidence versus speculation, and says "I need more information" when the data runs out. Current benchmarks heavily test the first skill but rarely test the latter two. This benchmark evaluates all three in a clinical-assessment-style setup where models must work from limited information, focusing on confidence calibration as a key marker of genuine metacognitive awareness. Congratulations, Qing! Social Cognition 11) HedgeDecode by Patricia Ji HedgeDecode measures whether a model can infer the socially correct response when intent is hedged, face-saving, or strategically implicit, and then execute that policy without naming the user's protective framing. This targets the heart of social cognition: understanding not just what people say, but what they mean, and responding in a way that respects rather than exposes their face-saving strategy. Congratulations, Patricia! 12) AdvisorBench: Do AI Models Widen the Advisory Divide? by Ana Pirtac "Hey Gemini, you think I should go to college?" That question is being asked right now by a first-generation student whose parents never set foot on a campus. AdvisorBench tests whether AI advisors give worse guidance to users who communicate at lower literacy levels, measuring whether the quality, depth, and actionability of advice degrades based on communication style. A critically important social cognition benchmark with real-world implications for equity. Congratulations, Ana! Attention 13) RIAC: Repetition-Induced Attention Collapse by Shankha Suvro Mullick Standard AI benchmarks test what models know. RIAC tests how well they attend. A model that correctly extracts a value from a clean sentence may completely fail when that same sentence is surrounded by repeated distractor tokens. No existing benchmark systematically isolates this failure mode, making RIAC the first to measure whether attention mechanisms degrade under repetition-based interference. Congratulations, Shankha! 14) ABC: Attention Benchmark for Cognition by Patrycja Węgrzynowicz and Ksawery Kopeć ABC evaluates selective attention under interference across both text and visual modalities, distinguishing between feature-sensitive attention and structure-sensitive attention inspired by Gestalt psychology principles. Testing 15 models across 2,160 examples, the benchmark found that selective attention is not monolithic: models that perform well matching explicit local features often fail when the correct answer depends on identifying the right group, region, or structural unit. Congratulations, Patrycja and Ksawery! Congratulations to all of the winners! Nick, on behalf of the Measuring Progress Toward AGI hackathon organizers