SWE-bench 是一个 AI 评测基准，用于评估模型完成真实软件工程任务的能力。
我们最新的模型，升级版 Claude 3.5 Sonnet，在软件工程评测 SWE-bench Verified 上取得了 49% 的成绩，超越了此前最优模型 45% 的表现。本文解释了我们在该模型周围构建的“智能体”，旨在帮助开发者从 Claude 3.5 Sonnet 中获得最佳性能。
SWE-bench 是一个 AI 评测基准，用于评估模型完成真实软件工程任务的能力。具体来说，它测试模型如何解决来自热门开源 Python 仓库的 GitHub 问题。对于基准中的每项任务，AI 模型会获得一个配置好的 Python 环境，以及该问题被解决前一刻的仓库检出（本地工作副本）。然后，模型需要理解、修改并测试代码，最后提交其提出的解决方案。
每个解决方案都会根据关闭原始 GitHub 问题的拉取请求中的真实单元测试进行评分。这用于测试 AI 模型是否能够实现与原始 PR 人类作者相同的功能。
SWE-bench 不仅孤立地评估 AI 模型，而是评估整个“智能体”系统。在此语境下，“智能体”指的是 AI 模型及其周围的软件脚手架的组合。这个脚手架负责生成输入模型的提示词、解析模型输出以执行操作，并管理交互循环——在该循环中，模型先前操作的结果会被整合到其下一个提示词中。即使使用相同的底层 AI 模型，智能体在 SWE-bench 上的表现也可能因脚手架的不同而有显著差异。
还有许多其他用于评估大语言模型编码能力的基准，但 SWE-bench 因以下几个原因而日益流行：
- 它使用来自实际项目的真实工程任务，而非竞赛或面试风格的问题；
- 这一领域尚未饱和——仍有很大的改进空间。目前还没有任何模型在 SWE-bench Verified 上的完成率超过 50%（不过，在撰写本文时，更新后的 Claude 3.5 Sonnet 已达到 49%）。
- 它衡量的是整个“智能体”，而非孤立的模型。开源开发者与初创公司在优化脚手架方面取得了巨大成功，显著提升了同一模型下的性能表现。
请注意，原始的 SWE-bench 数据集包含一些任务，这些任务如果没有 GitHub issue 之外的额外上下文（例如关于要返回的特定错误信息）是无法解决的。SWE-bench-Verified 是 SWE-bench 的一个包含 500 个问题的子集，经过人工审核以确保其可解性，因此能最清晰地衡量编码智能体的性能。本文后续将引用这一基准。
达到业界顶尖水平
工具使用型智能体
我们在创建针对更新版 Claude 3.5 Sonnet 优化的智能体脚手架时的设计理念是：尽可能将控制权交给大语言模型本身，并保持脚手架的最小化。该智能体包含一个提示词、一个用于执行 Bash 命令的 Bash 工具，以及一个用于查看和编辑文件及目录的编辑工具。我们会持续采样，直到模型自行判断任务已完成，或超出其 20 万 token 的上下文长度。这个脚手架允许模型自行判断如何解决问题，而不是被硬编码到特定的模式或工作流程中。
提示词为模型概述了一种建议的方法，但对于此任务而言，它并不过于冗长或详细。模型可以自由选择如何在步骤之间推进，而不是遵循严格且离散的转换。如果你对 token 数量不敏感，明确鼓励模型生成较长的回复会有所帮助。
以下代码展示了我们智能体脚手架中的提示词：
<uploaded_files>
{location}
</uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs). Consider the following PR description:
<pr_description>
{pr_description}
</pr_description>
Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!
Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.
Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize yourself with its structure.
2. Create a script to reproduce the error and execute it with `python <filename.py>` using the BashTool, to confirm the error
3. Edit the sourcecode of the repo to resolve the issue
4. Rerun your reproduce script and confirm that the error is fixed!
5. Think about edgecases and make sure your fix handles them as well
Your thinking should be thorough and so it's fine if it's very long.模型的第一个工具用于执行 Bash 命令。其模式很简单，仅接收要在环境中运行的命令。然而，该工具的描述更为重要。它包含了对模型更详细的指令，包括输入转义、无网络访问权限以及如何在后台运行命令。
接下来，我们展示 Bash 工具的规格说明：
{
"name": "bash",
"description": "Run commands in a bash shell\n
* When invoking this tool, the contents of the \"command\" parameter does NOT need to be XML-escaped.\n
* You don't have access to the internet via this tool.\n
* You do have access to a mirror of common linux and python packages via apt and pip.\n
* State is persistent across command calls and discussions with the user.\n
* To inspect a particular line range of a file, e.g. lines 10-25, try 'sed -n 10,25p /path/to/the/file'.\n
* Please avoid commands that may produce a very large amount of output.\n
* Please run long lived commands in the background, e.g. 'sleep 10 &' or start a server in the background.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "The bash command to run."
}
},
"required": ["command"]
}
}模型的第二个工具（编辑工具）要复杂得多，包含了模型查看、创建和编辑文件所需的一切。同样，我们的工具描述中包含了关于如何使用该工具的详细信息。
我们在各种智能体任务中为这些工具的描述和规格投入了大量精力。我们进行了测试，以发现模型可能误解规格的任何方式，或使用工具时可能存在的陷阱，然后修改描述以预先避免这些问题。我们认为，应该像为人类设计工具界面投入大量关注一样，为模型设计工具界面投入更多关注。
以下代码展示了我们编辑工具的描述：
{
"name": "str_replace_editor",
"description": "Custom editing tool for viewing, creating and editing files\n
* State is persistent across command calls and discussions with the user\n
* If `path` is a file, `view` displays the result of applying `cat -n`. If `path` is a directory, `view` lists non-hidden files and directories up to 2 levels deep\n
* The `create` command cannot be used if the specified `path` already exists as a file\n
* If a `command` generates a long output, it will be truncated and marked with `<response clipped>` \n
* The `undo_edit` command will revert the last edit made to the file at `path`\n
\n
Notes for using the `str_replace` command:\n
* The `old_str` parameter should match EXACTLY one or more consecutive lines from the original file. Be mindful of whitespaces!\n
* If the `old_str` parameter is not unique in the file, the replacement will not be performed. Make sure to include enough context in `old_str` to make it unique\n
* The `new_str` parameter should contain the edited lines that should replace the `old_str`",
...我们提升性能的一种方法是让工具"防错"。例如，当智能体移出根目录后，模型有时可能会搞乱相对文件路径。为了防止这种情况，我们简单地让工具始终要求使用绝对路径。
我们尝试了几种不同的策略来指定对现有文件的编辑，其中字符串替换的可靠性最高，即模型在给定文件中指定用 `new_str` 替换 `old_str`。只有当 `old_str` 恰好匹配一次时，替换才会执行。如果匹配次数多于或少于一次，模型会看到相应的错误消息以便重试。
我们的编辑工具规格说明如下所示：
...
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "undo_edit"],
"description": "The commands to run. Allowed options are: `view`, `create`, `str_replace`, `insert`, `undo_edit`."
},
"file_text": {
"description": "Required parameter of `create` command, with the content of the file to be created.",
"type": "string"
},
"insert_line": {
"description": "Required parameter of `insert` command. The `new_str` will be inserted AFTER the line `insert_line` of `path`.",
"type": "integer"
},
"new_str": {
"description": "Required parameter of `str_replace` command containing the new string. Required parameter of `insert` command containing the string to insert.",
"type": "string"
},
"old_str": {
"description": "Required parameter of `str_replace` command containing the string in `path` to replace.",
"type": "string"
},
"path": {
"description": "Absolute path to file or directory, e.g. `/repo/file.py` or `/repo`.",
"type": "string"
},
"view_range": {
"description": "Optional parameter of `view` command when `path` points to a file. If none is given, the full file is shown. If provided, the file will be shown in the indicated line number range, e.g. [11, 12] will show lines 11 and 12. Indexing at 1 to start. Setting `[start_line, -1]` shows all lines from `start_line` to the end of the file.",
"items": {
"type": "integer"
},
"type": "array"
}
},
"required": ["command", "path"]
}
}结果
总体而言，升级后的 Claude 3.5 Sonnet 在推理、编码和数学能力上均优于我们之前的模型以及之前的顶尖模型。它还展现了更强的智能体能力：这些工具和脚手架帮助将这些提升的能力发挥到最佳效果。
| 模型 | Claude 3.5 Sonnet（新版） | 之前的顶尖模型 | Claude 3.5 Sonnet（旧版） | Claude 3 Opus |
|---|---|---|---|---|
| SWE-bench Verified 得分 | 49% | 45% | 33% | 22% |
我们部分模型在 SWE-bench Verified 上的得分，均使用此智能体脚手架。
智能体行为示例
运行该基准测试时，我们以 SWE-Agent 框架为基础构建智能体代码。在下面的日志中，我们将智能体的文本输出、工具调用及工具响应分别呈现为 THOUGHT、ACTION 和 OBSERVATION，尽管我们并未强制要求模型遵循固定的输出顺序。
以下代码块将逐步展示 Sonnet 3.5 解决 SWE-bench 问题的典型流程。
在第一个代码块中，你可以看到模型收到的初始提示词片段，其中 `{pr_description}` 已被替换为 SWE-bench 任务中的实际内容。重要的是，该任务包含了复现问题的步骤，这为模型提供了有价值的调查起点。
<uploaded_files>
/repo
</uploaded_files>
I've uploaded a python code repository in the directory /repo (not in /tmp/inputs). Consider the following PR description:
<pr_description>
linear_model.RidgeClassifierCV's Parameter store_cv_values issue
#### Description
Parameter store_cv_values error on sklearn.linear_model.RidgeClassifierCV
#### Steps/Code to Reproduce
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas = np.arange(0.1, 1000, 0.1), normalize = True,
store_cv_values = True).fit(x, y)
#### Expected Results
Expected to get the usual ridge regression model output, keeping the cross validation predictions as attribute.
#### Actual Results
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'
lm.RidgeClassifierCV actually has no parameter store_cv_values, even though some attributes depends on it.模型做出响应，首先使用编辑工具查看仓库结构。你可以在 THOUGHT 和 ACTION 下看到模型的文本输出及工具调用参数，在 OBSERVATION 下看到部分工具输出：
THOUGHT:
I'll help you implement the changes to satisfy the PR description. Let's follow the steps you outlined.
1. First, let's explore the repo structure:
ACTION (str_replace_editor):
command:
view
path:
/repo
OBSERVATION:
Here's the files and directories up to 2 levels deep in /repo, excluding hidden items:
/repo
/repo/PULL_REQUEST_TEMPLATE.md
/repo/ISSUE_TEMPLATE.md
/repo/sklearn
/repo/sklearn/decomposition
/repo/sklearn/kernel_ridge.py
/repo/sklearn/model_selection
/repo/sklearn/covariance
/repo/sklearn/cluster
/repo/sklearn/base.py
/repo/sklearn/dummy.py
/repo/sklearn/naive_bayes.py
...现在模型对仓库结构有了更清晰的理解，它使用编辑工具创建了一个新脚本，用于复现问题并测试其修复方案：
THOUGHT:
Based on the repository structure, we'll need to modify the Ridge Classifier in the `sklearn/linear_model` directory.
Let's create a script to reproduce the error:
ACTION (str_replace_editor):
command:
create
path:
/repo/reproduce_error.py
file_text:
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas=np.arange(0.1, 1000, 0.1), normalize=True,
store_cv_values=True).fit(x, y)
OBSERVATION:
File created successfully at: /repo/reproduce_error.py随后模型使用 Bash 工具执行自己编写的脚本，并成功复现了任务中的问题：
THOUGHT:
Let's verify that we can reproduce the error:
ACTION (bash):
command:
python3 /repo/reproduce_error.py
OBSERVATION:
Traceback (most recent call last):
File "/repo/reproduce_error.py", line 10, in <module>
store_cv_values=True).fit(x, y)
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'接下来，模型使用编辑工具修改仓库中的源代码，并重新运行脚本以验证修改是否解决了问题：
THOUGHT:
The error is reproducible. Looking at the code, I can see that the issue is that while the `RidgeClassifierCV` class inherits from `_BaseRidgeCV`, it fails to pass the `store_cv_values` parameter from its constructor through to the base class constructor.
Let's fix this:
ACTION (str_replace_editor):
command:
str_replace
path:
/repo/sklearn/linear_model/ridge.py
old_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv)
self.class_weight = class_weight
new_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None,
store_cv_values=False):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv, store_cv_values=store_cv_values)
self.class_weight = class_weight在这个具体示例中，模型经过 12 步操作后决定提交结果。随后任务测试成功运行，验证了模型的解决方案确实解决了问题。部分任务需要模型经过 100 多轮交互才提交解决方案；而在另一些任务中，模型会持续尝试直至上下文耗尽。
通过对比更新版 Claude 3.5 Sonnet 与旧版模型的尝试过程，我们发现更新版 3.5 Sonnet 更频繁地进行自我修正。它还展现出尝试多种不同解决方案的能力，而非陷入反复犯同一错误的困境。
挑战
SWE-bench Verified 是一个强大的评估工具，但其运行复杂度远高于简单的单轮评估。以下是我们使用该工具时遇到的一些挑战——其他 AI 开发者也可能面临同样的问题。
- 持续时间和高昂的 token 成本。上述示例来自一个成功在 12 步内完成的案例。然而，许多成功的运行需要模型进行数百轮交互，并消耗超过 10 万个 token。升级后的 Claude 3.5 Sonnet 非常执着：只要有足够的时间，它通常能找到解决问题的方法，但这可能成本高昂；
- 评分。在检查失败任务时，我们发现了一些情况：模型行为正确，但存在环境设置问题，或者安装补丁被应用了两次。解决这些系统性问题对于准确了解 AI 智能体的性能至关重要。
- 隐藏测试。由于模型无法看到用于评分的测试，它常常在任务实际失败时“认为”自己成功了。其中一些失败是因为模型在错误的抽象层级上解决了问题（应用了临时补丁而非进行更深层的重构）。另一些失败则感觉不太公平：它们解决了问题，但与原始任务中的单元测试不匹配。
- 多模态。尽管升级后的 Claude 3.5 Sonnet 拥有出色的视觉和多模态能力，但我们并未实现让其查看保存到文件系统或通过 URL 引用的文件的功能。这使得调试某些任务（尤其是来自 Matplotlib 的任务）变得特别困难，并且容易导致模型幻觉。对于开发者来说，这显然存在一些唾手可得的改进空间——SWE-bench 已经推出了一个专注于多模态任务的新评估。我们期待看到开发者们在不久的将来使用 Claude 在此评估上取得更高分数。
升级后的 Claude 3.5 Sonnet 在 SWE-bench Verified 上达到了 49% 的成绩，凭借一个简单的提示词和两个通用工具，超越了之前的最优水平（45%）。我们相信，使用新版 Claude 3.5 Sonnet 进行开发的开发者们，将很快在我们最初展示的基础上，找到新的、更好的方法来提升 SWE-bench 分数。
致谢
Erik Schluntz 优化了 SWE-bench 智能体并撰写了这篇博文。Simon Biggs、Dawn Drain 和 Eric Christiansen 协助实现了该基准测试。Shauna Kravec、Dawn Drain、Felipe Rosso、Nova DasSarma、Ven Chandrasekaran 以及许多其他人为训练 Claude 3.5 Sonnet 在智能体编码方面表现出色做出了贡献。
SWE-bench is an AI evaluation benchmark that assesses a model's ability to complete real-world software engineering tasks.
Our latest model, the upgraded Claude 3.5 Sonnet, achieved 49% on SWE-bench Verified, a software engineering evaluation, beating the previous state-of-the-art model's 45%. This post explains the "agent" we built around the model, and is intended to help developers get the best possible performance out of Claude 3.5 Sonnet.
SWE-bench is an AI evaluation benchmark that assesses a model's ability to complete real-world software engineering tasks. Specifically, it tests how the model can resolve GitHub issues from popular open-source Python repositories. For each task in the benchmark, the AI model is given a set up Python environment and the checkout (a local working copy) of the repository from just before the issue was resolved. The model then needs to understand, modify, and test the code before submitting its proposed solution.
Each solution is graded against the real unit tests from the pull request that closed the original GitHub issue. This tests whether the AI model was able to achieve the same functionality as the original human author of the PR.
SWE-bench doesn't just evaluate the AI model in isolation, but rather an entire "agent" system. In this context, an "agent" refers to the combination of an AI model and the software scaffolding around it. This scaffolding is responsible for generating the prompts that go into the model, parsing the model's output to take action, and managing the interaction loop where the result of the model's previous action is incorporated into its next prompt. The performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.
There are many other benchmarks for the coding abilities of Large Language Models, but SWE-bench has gained in popularity for several reasons:
- It uses real engineering tasks from actual projects, rather than competition- or interview-style questions;
- It is not yet saturated—there’s plenty of room for improvement. No model has yet crossed 50% completion on SWE-bench Verified (though the updated Claude 3.5 Sonnet is, at the time of writing, at 49%);
- It measures an entire "agent", rather than a model in isolation. Open-source developers and startups have had great success in optimizing scaffoldings to greatly improve the performance around the same model.
Note that the original SWE-bench dataset contains some tasks that are impossible to solve without additional context outside of the GitHub issue (for example, about specific error messages to return). SWE-bench-Verified is a 500 problem subset of SWE-bench that has been reviewed by humans to make sure they are solvable, and thus provides the most clear measure of coding agents' performance. This is the benchmark to which we’ll refer in this post.
Achieving state-of-the-art
Tool Using Agent
Our design philosophy when creating the agent scaffold optimized for updated Claude 3.5 Sonnet was to give as much control as possible to the language model itself, and keep the scaffolding minimal. The agent has a prompt, a Bash Tool for executing bash commands, and an Edit Tool, for viewing and editing files and directories. We continue to sample until the model decides that it is finished, or exceeds its 200k context length. This scaffold allows the model to use its own judgment of how to pursue the problem, rather than be hardcoded into a particular pattern or workflow.
The prompt outlines a suggested approach for the model, but it’s not overly long or too detailed for this task. The model is free to choose how it moves from step to step, rather than having strict and discrete transitions. If you are not token-sensitive, it can help to explicitly encourage the model to produce a long response.
The following code shows the prompt from our agent scaffold:
<uploaded_files>
{location}
</uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs). Consider the following PR description:
<pr_description>
{pr_description}
</pr_description>
Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!
Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.
Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize yourself with its structure.
2. Create a script to reproduce the error and execute it with `python <filename.py>` using the BashTool, to confirm the error
3. Edit the sourcecode of the repo to resolve the issue
4. Rerun your reproduce script and confirm that the error is fixed!
5. Think about edgecases and make sure your fix handles them as well
Your thinking should be thorough and so it's fine if it's very long.The model's first tool executes Bash commands. The schema is simple, taking only the command to be run in the environment. However, the description of the tool carries more weight. It includes more detailed instructions for the model, including escaping inputs, lack of internet access, and how to run commands in the background.
Next, we show the spec for the Bash Tool:
{
"name": "bash",
"description": "Run commands in a bash shell\n
* When invoking this tool, the contents of the \"command\" parameter does NOT need to be XML-escaped.\n
* You don't have access to the internet via this tool.\n
* You do have access to a mirror of common linux and python packages via apt and pip.\n
* State is persistent across command calls and discussions with the user.\n
* To inspect a particular line range of a file, e.g. lines 10-25, try 'sed -n 10,25p /path/to/the/file'.\n
* Please avoid commands that may produce a very large amount of output.\n
* Please run long lived commands in the background, e.g. 'sleep 10 &' or start a server in the background.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "The bash command to run."
}
},
"required": ["command"]
}
}The model's second tool (the Edit Tool) is much more complex, and contains everything the model needs for viewing, creating, and editing files. Again, our tool description contains detailed information for the model about how to use the tool.
We put a lot of effort into the descriptions and specs for these tools across a wide variety of agentic tasks. We tested them to uncover any ways that the model might misunderstand the spec, or the possible pitfalls of using the tools, then edited the descriptions to preempt these problems. We believe that much more attention should go into designing tool interfaces for models, in the same way that a large amount of attention goes into designing tool interfaces for humans.
The following code shows the description for our Edit Tool:
{
"name": "str_replace_editor",
"description": "Custom editing tool for viewing, creating and editing files\n
* State is persistent across command calls and discussions with the user\n
* If `path` is a file, `view` displays the result of applying `cat -n`. If `path` is a directory, `view` lists non-hidden files and directories up to 2 levels deep\n
* The `create` command cannot be used if the specified `path` already exists as a file\n
* If a `command` generates a long output, it will be truncated and marked with `<response clipped>` \n
* The `undo_edit` command will revert the last edit made to the file at `path`\n
\n
Notes for using the `str_replace` command:\n
* The `old_str` parameter should match EXACTLY one or more consecutive lines from the original file. Be mindful of whitespaces!\n
* If the `old_str` parameter is not unique in the file, the replacement will not be performed. Make sure to include enough context in `old_str` to make it unique\n
* The `new_str` parameter should contain the edited lines that should replace the `old_str`",
...One way we improved performance was to "error-proof" our tools. For instance, sometimes models could mess up relative file paths after the agent had moved out of the root directory. To prevent this, we simply made the tool always require an absolute path.
We experimented with several different strategies for specifying edits to existing files and had the highest reliability with string replacement, where the model specifies `old_str` to replace with `new_str` in the given file. The replacement will only occur if there is exactly one match of `old_str`. If there are more or fewer matches, the model is shown an appropriate error message for it to retry.
The spec for our Edit Tool is shown below:
...
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "undo_edit"],
"description": "The commands to run. Allowed options are: `view`, `create`, `str_replace`, `insert`, `undo_edit`."
},
"file_text": {
"description": "Required parameter of `create` command, with the content of the file to be created.",
"type": "string"
},
"insert_line": {
"description": "Required parameter of `insert` command. The `new_str` will be inserted AFTER the line `insert_line` of `path`.",
"type": "integer"
},
"new_str": {
"description": "Required parameter of `str_replace` command containing the new string. Required parameter of `insert` command containing the string to insert.",
"type": "string"
},
"old_str": {
"description": "Required parameter of `str_replace` command containing the string in `path` to replace.",
"type": "string"
},
"path": {
"description": "Absolute path to file or directory, e.g. `/repo/file.py` or `/repo`.",
"type": "string"
},
"view_range": {
"description": "Optional parameter of `view` command when `path` points to a file. If none is given, the full file is shown. If provided, the file will be shown in the indicated line number range, e.g. [11, 12] will show lines 11 and 12. Indexing at 1 to start. Setting `[start_line, -1]` shows all lines from `start_line` to the end of the file.",
"items": {
"type": "integer"
},
"type": "array"
}
},
"required": ["command", "path"]
}
}Results
In general, the upgraded Claude 3.5 Sonnet demonstrates higher reasoning, coding, and mathematical abilities than our prior models, and the previous state-of-the-art model. It also demonstrates improved agentic capabilities: the tools and scaffolding help put those improved abilities to their best use.
| Model | Claude 3.5 Sonnet (new) | Previous SOTA | Claude 3.5 Sonnet (old) | Claude 3 Opus |
|---|---|---|---|---|
| SWE-bench Verified score | 49% | 45% | 33% | 22% |
Scores on SWE-bench Verified for some of our models, all using this agent scaffold.
Examples of agent behavior
For running the benchmark, we used the SWE-Agent framework as a foundation for our agent code. In our logs below, we render the agent's text output, tool calls, and tool responses as THOUGHT, ACTION, and OBSERVATION, even though we don’t constrain the model to a fixed ordering.
The code blocks below will walk through a typical case of the Sonnet 3.5 solving a SWE-bench problem.
In this first block, you can see part of the initial prompt given to the model, with `{pr_description}` filled in with the real value from a SWE-bench task. Importantly, this task contains steps to reproduce the issue, which will give the model a valuable starting point to investigate.
<uploaded_files>
/repo
</uploaded_files>
I've uploaded a python code repository in the directory /repo (not in /tmp/inputs). Consider the following PR description:
<pr_description>
linear_model.RidgeClassifierCV's Parameter store_cv_values issue
#### Description
Parameter store_cv_values error on sklearn.linear_model.RidgeClassifierCV
#### Steps/Code to Reproduce
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas = np.arange(0.1, 1000, 0.1), normalize = True,
store_cv_values = True).fit(x, y)
#### Expected Results
Expected to get the usual ridge regression model output, keeping the cross validation predictions as attribute.
#### Actual Results
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'
lm.RidgeClassifierCV actually has no parameter store_cv_values, even though some attributes depends on it.The model responds and first uses the Edit Tool to view the repository structure. You can see the model's text output and tool call arguments under THOUGHT and ACTION, and part of the tool's output under OBSERVATION:
THOUGHT:
I'll help you implement the changes to satisfy the PR description. Let's follow the steps you outlined.
1. First, let's explore the repo structure:
ACTION (str_replace_editor):
command:
view
path:
/repo
OBSERVATION:
Here's the files and directories up to 2 levels deep in /repo, excluding hidden items:
/repo
/repo/PULL_REQUEST_TEMPLATE.md
/repo/ISSUE_TEMPLATE.md
/repo/sklearn
/repo/sklearn/decomposition
/repo/sklearn/kernel_ridge.py
/repo/sklearn/model_selection
/repo/sklearn/covariance
/repo/sklearn/cluster
/repo/sklearn/base.py
/repo/sklearn/dummy.py
/repo/sklearn/naive_bayes.py
...Now that the model has a better understanding of the repository structure, it uses the Edit Tool to create a new script that it will use to reproduce the issue and test its fix:
THOUGHT:
Based on the repository structure, we'll need to modify the Ridge Classifier in the `sklearn/linear_model` directory.
Let's create a script to reproduce the error:
ACTION (str_replace_editor):
command:
create
path:
/repo/reproduce_error.py
file_text:
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas=np.arange(0.1, 1000, 0.1), normalize=True,
store_cv_values=True).fit(x, y)
OBSERVATION:
File created successfully at: /repo/reproduce_error.pyThe model then uses the Bash Tool to execute the script it wrote, and successfully reproduces the issue from the task:
THOUGHT:
Let's verify that we can reproduce the error:
ACTION (bash):
command:
python3 /repo/reproduce_error.py
OBSERVATION:
Traceback (most recent call last):
File "/repo/reproduce_error.py", line 10, in <module>
store_cv_values=True).fit(x, y)
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'From here on, the model uses the Edit Tool to change the source code in the repository and reruns its script to verify whether the change has resolved the issue:
THOUGHT:
The error is reproducible. Looking at the code, I can see that the issue is that while the `RidgeClassifierCV` class inherits from `_BaseRidgeCV`, it fails to pass the `store_cv_values` parameter from its constructor through to the base class constructor.
Let's fix this:
ACTION (str_replace_editor):
command:
str_replace
path:
/repo/sklearn/linear_model/ridge.py
old_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv)
self.class_weight = class_weight
new_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None,
store_cv_values=False):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv, store_cv_values=store_cv_values)
self.class_weight = class_weightIn this particular example, the model worked for 12 steps before deciding that it was ready to submit. The task's tests then ran successfully, verifying that the model's solution addressed the problem. Some tasks took more than 100 turns before the model submitted its solution; in others, the model kept trying until it ran out of context.
From reviewing attempts from the updated Claude 3.5 Sonnet compared to older models, updated 3.5 Sonnet self-corrects more often. It also shows an ability to try several different solutions, rather than getting stuck making the same mistake over and over.
Challenges
SWE-bench Verified is a powerful evaluation, but it’s also more complex to run than simple, single-turn evals. These are some of the challenges that we faced in using it—challenges that other AI developers might also encounter.
- Duration and high token costs. The examples above are from a case that was successfully completed in 12 steps. However, many successful runs took hundreds of turns for the model to resolve, and >100k tokens. The updated Claude 3.5 Sonnet is tenacious: it can often find its way around a problem given enough time, but that can be expensive;
- Grading. While inspecting failed tasks, we found cases where the model behaved correctly, but there were environment setup issues, or problems with install patches being applied twice. Resolving these systems issues is crucial for getting an accurate picture of an AI agent's performance.
- Hidden tests. Because the model cannot see the tests it's being graded against, it often “thinks” that it has succeeded when the task actually is a failure. Some of these failures are because the model solved the problem at the wrong level of abstraction (applying a bandaid instead of a deeper refactor). Other failures feel a little less fair: they solve the problem, but do not match the unit tests from the original task.
- Multimodal. Despite the updated Claude 3.5 Sonnet having excellent vision and multimodal capabilities, we did not implement a way for it to view files saved to the filesystem or referenced as URLs. This made debugging certain tasks (especially those from Matplotlib) especially difficult, and also prone to model hallucinations. There is definitely low-hanging fruit here for developers to improve upon—and SWE-bench has launched a new evaluation focused on multi-modal tasks. We look forward to seeing developers achieve higher scores on this eval with Claude in the near future.
The upgraded Claude 3.5 Sonnet achieved 49% on SWE-bench Verified, beating the previous state-of-the-art (45%), with a simple prompt and two general purpose tools. We feel confident that developers building with the new Claude 3.5 Sonnet will quickly find new, better ways to improve SWE-bench scores over what we've initially demonstrated here.
Acknowledgements
Erik Schluntz optimized the SWE-bench agent and wrote this blog post. Simon Biggs, Dawn Drain, and Eric Christiansen helped implement the benchmark. Shauna Kravec, Dawn Drain, Felipe Rosso, Nova DasSarma, Ven Chandrasekaran, and many others contributed to training Claude 3.5 Sonnet to be excellent at agentic coding.