AgentSearchBench： 一个面向真实场景的AI智能体搜索基准
这篇论文给火热的Agent生态泼了盆冷水，它用近万个真实Agent证明，靠描述找Agent基本不靠谱。做Agent平台或搜索的开发者，这是你必须面对的基准和改进方向。
研究团队推出了AgentSearchBench，这是一个用于评估真实场景中AI智能体搜索能力的大规模基准。该基准从多个平台收集了近10,000个真实世界智能体，将智能体搜索形式化为可执行任务查询和高级任务描述下的检索与重排序问题，并采用基于执行结果的性能信号来评估相关性。实验表明，语义相似性与智能体实际性能之间存在持续差距，暴露了仅依赖描述进行检索和重排序方法的局限性。研究进一步证明，轻量级的行为信号（包括执行感知探测）能显著提升排序质量，凸显了将执行信号纳入智能体发现过程的重要性。相关代码已开源。
Computer Science > Artificial Intelligence
Title:AgentSearchBench: A Benchmark for AI Agent Search in the Wild
View PDF HTML (experimental)Abstract:The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2604.22436 [cs.AI] |
| (or arXiv:2604.22436v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.22436 arXiv-issued DOI via DataCite |
Current browse context:
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
- Author
- Venue
- Institution
- Topic
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.