CODA-BENCH:代码智能体能处理数据密集型任务吗?
阅读原文· arxiv.orgCODA-BENCH 是首个同时评估代码智能体在代码与数据两方面能力的基准测试,专为数据密集型环境设计。该基准基于 Kaggle 生态系统搭建,包含数百个数据集,共 1009 个任务,覆盖 31 个社区。每个任务环境平均拥有 980 个文件,模拟真实的数据规模与噪声。对现有高级智能体的评估显示,即使表现最佳的系统也难以有效整合数据发现与代码执行,成功率仅为 61.1%,暴露了当前智能体在处理数据密集型任务时的能力缺口。
Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.