WorldReward 提出面向相机条件世界模型的 VLM 奖励建模方法

HuggingFace Daily Papers(社区热门论文)·2026-09-03 08:00·1天前
AI 导读

WorldReward 是一个基于 VLM 的成对偏好奖励模型,为相机条件世界模型统一评估动作一致性与视觉质量。方法将视频分解为动作对齐的块并投票聚合,配套发布人工标注的 WorldReward-Bench,在三个维度上分别超过 GPT-5.5 约 3.42、1.45 和 3.56 个百分点,用于 HY-WorldPlay 1.5 的 RL 后训练可同时提升动作执行与视觉质量。

HuggingFace Daily Papers(社区热门论文)
40AI 编辑部评分,满分 100

WorldReward 提出面向相机条件世界模型的 VLM 奖励建模方法

2026-09-03 08:00· 1天前
AI 导读

WorldReward 是一个基于 VLM 的成对偏好奖励模型,为相机条件世界模型统一评估动作一致性与视觉质量。方法将视频分解为动作对齐的块并投票聚合,配套发布人工标注的 WorldReward-Bench,在三个维度上分别超过 GPT-5.5 约 3.42、1.45 和 3.56 个百分点,用于 HY-WorldPlay 1.5 的 RL 后训练可同时提升动作执行与视觉质量。

Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess these requirements separately: geometry-based rewards estimate trajectory execution but cannot judge the visual quality of the executed motion, whereas image-based rewards measure frame quality without capturing action execution or temporal dynamics. We posit that a vision-language model (VLM) offers a shared reasoning space for relating actions to their visual outcomes. However, judging a complete long video against its full action sequence creates a lengthy, noisy context in which short-lived local action evidence can be missed or diluted. We present WorldReward, a VLM-based pairwise preference reward model that unifies action-consistency and visual-quality evaluation for camera-conditioned world models. WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences. To train it, we construct a large-scale reasoning-augmented preference dataset using structured judgments generated by a frontier VLM and refined through tool-based agent auditing and targeted human review. We further introduce WorldReward-Bench, a human-annotated benchmark measuring reward-model agreement with human preferences across action consistency, appearance quality, and motion quality. WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively. When used for RL post-training of HY-WorldPlay 1.5, it consistently improves both action execution and visual quality across short- to long-term horizons.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org