SWE-Together: Evaluating Coding Agents in Interactive User Sessions
阅读原文· arxiv.org现有编码智能体基准多为静态,仅凭最终代码评判。SWE-Together 从 11,260 条真实用户-智能体编码会话中筛选出 109 个仓库级任务,构建多轮交互基准。研究团队利用基于 LLM 的用户模拟器保留原始用户意图,并在智能体需要时提供反馈。评估同时衡量最终仓库正确性和交互中的纠正反馈次数。实验表明,更强智能体成功率更高且所需干预更少,预示用户体验提升。
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.