# SafeDiffusion-R1：面向安全扩散后训练的在线奖励引导

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-18 08:00
- AIHOT 分数：65
- AIHOT 链接：https://aihot.virxact.com/items/cmpcekovh01p7slaezztoeuq0
- 原文链接：https://arxiv.org/abs/2605.18719

## AI 摘要

本研究针对扩散模型去除预训练阶段不安全内容时面临的监督数据昂贵及离线方法易导致灾难性遗忘等问题，提出了SafeDiffusion-R1在线强化学习框架。该框架采用组相对策略优化算法，在正负文本提示上进行后训练，无需依赖配对监督数据。其核心在于引入了一种奖励引导机制，直接利用CLIP嵌入特性，在嵌入空间中引导文本表征向安全方向优化，从而无需微调专门的安全奖励模型。实验表明，该方法将不安全内容生成比例从48.9%降至18.07%，裸体检测数大幅减少，同时提升了组合生成质量，并可泛化至多种危害类别，达到当前最优水平。

## 正文

Diffusion models have been widely studied for removing unsafe content learned during pre-training. Existing methods require expensive supervised data, either unsafe-text paired with safe-image groundtruth or negative/positive image pairs, making them impractical to scale. Furthermore, offline reinforcement learning and supervised fine-tuning approaches that generate synthetic data offline suffer from catastrophic forgetting, degrading generation quality. We propose a novel online reinforcement learning framework that addresses both data scarcity and model degradation through post-training with Group Relative Policy Optimization (GRPO) on both negative and positive text prompts. To eliminate the need for fine-tuning specialized safe/unsafe reward models, we introduce a steering reward mechanism that exploits an inherent property of CLIP embeddings: steering text representations toward positive safety directions and away from negative ones in the embedding space. Our online-policy approach enables the model to learn from diverse prompts, including explicit unsafe content, without catastrophic forgetting. Extensive experiments demonstrate that our method reduces inappropriate content to 18.07\% (vs. 48.9\% for SD v1.4) and nudity detections to 15 (vs. 646 baseline) while improving compositional generation quality from 42.08\% to 47.83\% on GenEval. Remarkably, these safety gains generalize to out-of-domain unsafe prompts across seven harm categories, achieving state-of-the-art performance without supervised paired data or reward tuning. Github: https://github.com/MAXNORM8650/SafeDiffusion-R1.