通过一项详细审计，我们发现 SWE-Bench Pro 中存在广泛的任务问题，并估计约有 30% 的任务存在缺陷。
准确衡量我们模型的能力，对于做出合理的部署和安全决策至关重要，包括在 OpenAI 准备框架下的决策。每次发布新模型时，我们都会报告一系列外部和内部基准测试的结果，以追踪模型进展。当评估存在影响结果的缺陷时，它们可能会对能力产生错误的理解，歪曲安全案例并影响研究优先级。
我们最近调查了最广泛使用的编程基准之一 SWE-bench Verified，发现其存在根本性的设计和数据污染问题，并认为该评估已无法再为软件开发能力提供有意义的信号。当时，我们鼓励整个社区转向使用 SWE-Bench Pro。
SWE-Bench Pro 旨在改进 SWE-bench Verified，通过在更长周期和更真实的编程任务上测试模型，以更好地追踪智能体编程能力。与 SWE-bench Verified 类似，其任务是从一组公共和私有仓库的功能变更历史中通过程序化方式获取。模型需要实现一个解决方案，使其能通过新功能的测试，同时不破坏现有功能。在包含 731 个任务的公开子集中，前沿模型在八个月内通过率从 23.3% 提升到了 80.3%。
此后，我们对 SWE-Bench Pro 进行了类似的审计，使用一个数据点分析流程审查了该数据集。该流程审查了模型在任务上的尝试、任务元数据以及失败轨迹，以标记可能的评估缺陷。每个被标记的任务随后会经过多次调查智能体（investigator-agent）的遍历，并由五位经验丰富的软件工程师独立审查，存在分歧的情况会升级以进行进一步调查。
按问题类型标记的数据集占比
我们发现数据集中相当大一部分任务存在破坏性问题。我们的数据点分析流程标记了 200 个（27.4%）有缺陷的任务，而人工标注活动则识别出 249 个（34.1%）。
这些问题主要分为四类：
- 过于严格的测试会强制要求实现提示词中未明确指定的具体实现细节，从而导致许多功能正确的提交被判定为无效。
- 描述不充分的提示词遗漏了隐藏测试所强制要求、且无法合理推断出的需求。
- 覆盖率不足的测试对请求的功能检查不充分，因此不完整的修复也能通过测试。
- 具有误导性的提示词会将模型引向错误的行为方向，或与测试要求相矛盾。
我们的发现指出了策划困难但公平的基准测试的难度，以及智能体在可扩展的数据质量检查方面日益增长的实用性。基于这些结果，我们估计 SWE-bench Pro 中约有 30% 的任务存在缺陷，并建议模型开发者仔细审查结果。
方法论
我们的目标是确保任务失败反映的是模型的真实局限，而任务成功则代表对提示词要求的完整且有效的解决方案。为了检查评估所用数据的质量，我们创建了一个质量保证流程，用于评估每个数据点是否准确反映了模型能力。
一个初始的数据质量流程会标记出问题以供审查。我们通过更深度的、由智能体辅助的审计来验证被标记的任务，并开展了一项由经验丰富的工程师参与的人工标注工作。
一个初始的自动化过滤器会审查提供给模型的指令、模型解决任务的尝试，以及用于评判这些尝试的测试，以标记出可能存在缺陷或问题的示例。该过滤器标记了 286 个可能存在问题的任务。随后，我们通过两种方式对该子集进行了更深入的审查：一是人工监督的智能体审查，该审查使用调查智能体进行广泛检查，并最终由人工做出判断；二是与经验丰富的软件开发人员合作开展的人工标注工作。
人工监督的智能体审查
每个被标记的问题都会由基于 Codex 的调查智能体进行审计，这些智能体可以访问任务仓库和环境。这有助于它们区分合理的任务歧义（通常可以通过研究附近代码和仓库约定来解决）与真正的规范不足。该智能体可以运行测试、检查仓库中的文件，并调查模型在该任务上的尝试及其常见失败模式。在多次独立重复这些更深入的审计之后，一位研究人员审阅了摘要，做出最终判断，并标记了可能存在的问题。
人工标注活动
与此同时，我们对被标记的子集进行了一项人工标注活动。我们与经验丰富的软件工程师合作，他们在审阅任务之前接受了关于基准目标、问题分类和边缘案例的培训。每项任务由五位工程师审阅。
审阅者在利用流水线分析或记录作为辅助背景信息之前，会根据可见的问题陈述、测试用例和真实参考解决方案（称为黄金补丁）形成独立判断。然后，审阅者根据具体证据分配标签和严重等级，并将存在分歧或置信度较低的情况上报以供进一步审阅。
与调查智能体相比，人工审阅者更倾向于将任务标记为有缺陷。两种审阅路径在分类上也存在一些分歧，但在任何被标记的任务中，“无缺陷”都不是最常见的人工标签。在智能体流水线标记的分类中，审阅者的判断在 74% 的情况下是重叠的。
与智能体流水线相比，人工审阅者更倾向于为一项任务选择多个标签，这表明他们认为任务存在多种缺陷，或者无法干净地归入单一类别。这表明智能体加审阅者的流水线产生了保守的标注结果：它捕捉到了人工识别出的相同的主要失败模式，但同时低估了审阅者发现额外或重叠问题的情况。最大的差异出现在低覆盖率测试上，人工将其选为基准中 9.4% 任务的最常见问题，而智能体流水线的这一比例为 4.1%。
失败模式
在某些情况下，任务提示词规定了具体的实现方式，但隐藏测试用例却期望不同的行为。
OpenLibrary-77c16d5
此任务涉及规范化目录条目，并通过 `TocEntry.to_markdown()` 将其渲染回 Markdown 格式。任务提示词精确规定了字符级别的间距，描述了如何强制执行精确的空格和竖线，并给出了诸如 " | Chapter 1 | 1" 和 "** | Chapter 1 | 1" 的示例：
无
1"[空格]| Chapter 1 | 1"2"**[空格]| Chapter 1 | 1"3"[空格]| Just title | "
隐藏的 `test_to_markdown` 断言反而要求 " | Chapter 1 | 1" 和 "** | Chapter 1 | 1"：
无
1"[空格][空格]| Chapter 1 | 1"2"**[空格][空格]| Chapter 1 | 1"3"[空格][空格]| Just title | "
隐藏测试中有两个前导空格，但提供给模型的示例中仅包含一个前导空格。如果模型正确地遵循了给定的提示词，这一个字符的差异就会导致隐藏测试用例失败，该任务也会被标记为错误。
Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.
Accurately measuring our models’ capabilities is important for sound deployment and safety decisions, including decisions under OpenAI’s Preparedness Framework . With each model release, we report results for a variety of external and internal benchmarks to track model progress. When evaluations have flaws that affect results, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities.
We recently investigated how one of the most widely used coding benchmarks, SWE-bench Verified, had fundamental design and contamination issues, and found that the eval no longer provided meaningful signal on software development capabilities. At the time, we encouraged the wider community to switch to SWE-Bench Pro.
SWE-Bench Pro was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. As in SWE-bench Verified, tasks are sourced programmatically from the history of feature changes in a set of public and private repositories. Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months.
We’ve since performed a similar audit on SWE-Bench Pro, reviewing the dataset using a datapoint analysis pipeline. The pipeline reviewed model attempts at the task, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for further investigation.
Share of Dataset Flagged by Issue Type
We find evidence of breaking issues in a significant portion of the dataset. Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).
The issues primarily fell into four categories:
- Overly strict tests1 enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
- Underspecified prompts2 omit requirements that hidden tests enforce and that are not reasonably inferable.
- Low-coverage tests under check the requested feature, so incomplete fixes can pass.
- A misleading prompt points models toward the wrong behavior or contradicts what tests require.
Our findings point to the difficulty of curating hard but fair benchmarks and the growing utility of agents for scalable data quality checks. In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.
Methodology
Our aim is to ensure that task failures reflect genuine model limitations, and task successes reflect complete and valid solutions to the prompt requirements. To check the quality of the data used in the evaluation, we created a quality assurance pipeline to assess whether each datapoint accurately reflects model capabilities.
An initial data quality pipeline flags problems for review. We validate with a deeper agent-assisted audit of flagged tasks and a human annotation campaign working with experienced engineers.
An initial automated filter reviews the instructions given to the model, attempts by the model to solve the task, and the tests used to grade these attempts to flag likely broken or problematic examples. This filter flagged 286 potentially broken tasks. We then conducted a deeper review of that subset in two ways: a human-supervised agent review, which conducts extensive checks with investigator agents and a final human judgment; and a human annotation campaign working with experienced software developers.
Human-supervised agent review
Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment. This helps them distinguish reasonable task ambiguity, which can often be resolved by studying nearby code and repository conventions, from true underspecification. The agent can run tests, inspect files in the repo, and investigate model attempts and their common failure modes on the task. After several independent repeats of these deeper audits, a researcher reviewed the summaries, made a final judgment, and labeled the likely issues.
Human annotation campaign
In parallel, we ran a human annotation campaign over the flagged subset. We worked with experienced software engineers who were trained on the benchmark goals, issue taxonomy, and edge cases before reviewing tasks. Each task was reviewed by five engineers.
Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution (known as the gold patch) before using the pipeline analysis or transcript as supporting context. The reviewers then assigned a label and severity rating based on concrete evidence, and escalated disagreements or low-confidence cases for further review.
Human reviewers were more likely than the investigator agents to mark tasks as broken. There was also some disagreement on categories between the two review paths, but in no flagged task was “not broken” the most common human label. Of the categories the agent pipeline flagged, reviewers’ judgments overlapped in 74% of cases.
Compared with the agent pipeline, the human reviewers were also more likely to select multiple labels for a task, indicating that they found tasks to be broken in multiple ways or did not fit cleanly into a single category. This suggests the agent-plus-reviewer pipeline resulted in conservative labeling: it captured the same broad failure modes humans identified, while undercounting cases where reviewers saw additional or overlapping issues. The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.
Failure modes
In several cases the task prompt prescribed a specific implementation, but the hidden test cases expected different behavior.
OpenLibrary-77c16d5
This task involves normalizing table-of-contents entries and rendering them back to Markdown via TocEntry.to_markdown(). The task prompt specifies serialization down to character-level spacing, describing how exact spacing and pipes are enforced, and gives examples such as " | Chapter 1 | 1" and "** | Chapter 1 | 1":
None
1"[space]| Chapter 1 | 1"2"**[space]| Chapter 1 | 1"3"[space]| Just title | "
The hidden test_to_markdown assertions instead require " | Chapter 1 | 1" and "** | Chapter 1 | 1":
None
1"[space][space]| Chapter 1 | 1"2"**[space][space]| Chapter 1 | 1"3"[space][space]| Just title | "
There are two leading spaces in the hidden tests, but the example given to the model only contains one leading space. If a model rightly follows the given prompt, that one-character difference would fail the hidden test cases and the task would be marked incorrect.