Automatically patching an agent's harness is easy; keeping the patches that help is the hard part.

So if you let a model rewrite your prompts, tools, and hooks from failure traces, spend the rollout budget on a held-out split that catches collateral damage.

Harness optimization does work, but this paper finds that without a check on whether an update generalizes, the result lands below the hand-written harness it started from.

AutoSaddler diagnoses failed traces from a mini-batch, treats the harness as code, patches prompts, tools, and middleware, and keeps only updates that also improve a held-out development set.

If you're tuning an agent by hand or with an LLM in the loop, hold out a set of tasks that the patch was not written for, and score fixes minus regressions rather than fixes alone.

– arxiv. org/abs/2608.23041

Title: "AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces"