Adaptive Auto-Harness:面向开放任务流智能体系统部署的持续自改进框架
阅读原文· arxiv.org现有 Auto-Harness 系统仅针对固定离线基准评测,而开放任务流存在无终点历史、异构任务与分布偏移,导致单一密集更新装备性能先升后降。本文提出 Adaptive Auto-Harness,将距 oracle 装备差距分解为进化损失与适配损失,采用状态化多智能体进化器、带求解时路由的装备树及人类引导钩子来解决。在预测市场、安全竞赛与事件预测三个任务流上,该方法优于五个基线,消融实验验证了各模块贡献。代码已开源。
Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in https://github.com/A-EVO-Lab/AdaptiveHarness .