Executable Python code can define a vehicle that works on high-friction ground yet spins uselessly on ice. Can an agent revise that code and make it work again after the environment changes?

Introducing PACE-Bench, from @TsinghuaNLP. Each challenge starts with design code that scores 100 in a Source environment but fails after physical parameters mutate. The agent gets 20 sandbox interactions to adapt it to the new Target environment.

1⃣️ A hard, verifiable testbed. PACE-Bench contains 36 base tasks and 144 Source→Target pairs across six physics domains. The full benchmark compares ten self-evolving methods at three Qwen3 scales, with larger and frontier LLMs evaluated on selected subsets. The best full-benchmark result, Reflexion + Qwen3-14B, reaches 35.9% Pass@2. Scaling helps, but does not solve adaptation. On the 24-pair Statics subset, Qwen3-32B does not outperform Qwen3-14B under the same Vanilla protocol. GPT-5.5 reaches 66.7%, but still fails one third of the pairs. PACE-Bench remains far from saturated.

2⃣️ Fresh environment feedback matters more than simply “thinking more.” Revisions grounded in the latest simulator result are more reliable than multiple model-only edits before the next check. Effective self-evolution needs a tight loop of observing, revising, and verifying.

3⃣️ The core search challenge is balancing exploration and convergence. Reusing past experience can lock an agent onto an outdated design. Broader search can escape that design but wander without converging. A capable agent must do both: abandon invalid assumptions and focus on a workable mechanism.

4⃣️ More information does not remove the redesign bottleneck. On Statics + Kinematics, directly revealing every changed parameter still yields a lower best result than keeping exact changes hidden. An agent may know that friction decreased, yet not know whether to change geometry, control, or the entire mechanism. “Know what” is not “know how.”

PACE-Bench shows that reliable self-evolution requires grounded feedback, balanced search, and mechanism-level redesign. More scale, reflection, or information alone is not enough.

Project: https://thunlp.github.io/PACE-Bench/ Code: https://github.com/thunlp/PACE-Bench Paper: https://arxiv.org/pdf/2608.14441 X Threads: https://x.com/YuhaoZhan6/status/2089717175955407057 #AI #THUNLP #OpenBMB #LLM #Agents #SelfEvolving #DynamicEnvironments