# 多模态智能体推理的智能体探索性策略优化

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-27 08:00
- AIHOT 分数：66
- AIHOT 链接：https://aihot.virxact.com/items/cmpoxnk890a21slv44h3w53vj
- 原文链接：https://arxiv.org/abs/2605.28774

## AI 摘要

针对视觉语言模型在工具使用任务中存在的“思考-行动差距”（工具调用尝试率仅约30%，且其中约40%问题的所有工具调用均错误），研究提出AXPO（AI 智能体探索性策略优化）方法。该方法在标准强化学习（如GRPO）流程中，针对工具调用全错的子批次，固定思考前缀并重新采样工具调用及后续内容。在九个多模态基准和三种规模的Qwen3-VL-Thinking模型上，SFT+AXPO的平均性能优于SFT+GRPO（8B模型平均Pass@1和Pass@4均提升1.8个百分点）。8B的SFT+AXPO模型在Pass@4上甚至超越了参数量为其4倍的32B Base模型。

## 正文

Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.