HuggingFace Daily Papers(社区热门论文)
62AI 编辑部评分,满分 100

ToolArtist:面向智能体图像生成的工具使用统一多模态模型

2026-08-05 08:00· 1天前
AI 导读

ToolArtist 通过后训练统一多模态模型,将推理、外部工具调用与原生图像生成整合于单一智能体策略中。其采用监督微调配合搜索与图像生成工具,并引入 RAD-GRPO 强化学习算法,以意图与质量奖励联合优化。实验表明,全流程智能体控制方案持续优于固定流程或部分控制方案,训练数据与基础设施已开源。

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.

来源:HuggingFace Daily Papers(社区热门论文) · arxiv.org

ToolArtist:面向智能体图像生成的工具使用统一多模态模型

HuggingFace Daily Papers(社区热门论文)·2026-08-05 08:00·1天前
AI 导读

ToolArtist 通过后训练统一多模态模型,将推理、外部工具调用与原生图像生成整合于单一智能体策略中。其采用监督微调配合搜索与图像生成工具,并引入 RAD-GRPO 强化学习算法,以意图与质量奖励联合优化。实验表明,全流程智能体控制方案持续优于固定流程或部分控制方案,训练数据与基础设施已开源。

原文 · 保持原样,未翻译

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org