Physics-IQ Verified
阅读原文· arxiv.org本文对Physics-IQ基准进行系统审查,指出其缺陷并提出三项改进:优化提示词与真实数据质量以减少混淆因素,引入样本级评分系统使每个样本和指标权重相等。改进后的Physics-IQ Verified基准优化了57.6%的样本和34.8%的提示词。在六种图像到视频生成模型的对比研究中,排名出现中等但有意义的变化(Kendall's τ=0.46)。基准代码已开源。
Video generative models ( VGMs) have become a new frontier that can be used not just for video generation but for a multitude of downstream tasks, including world modeling. To advance these tasks, a good video model must understand the physical reality of the world. Evaluating this understanding is an emerging field and has led to the Physics-IQ benchmark, which quantifies this explicitly by comparing model-generated videos to real-world videos of physical experiments. In this work, we present a systematic audit of the Physics-IQ benchmark, expose shortcomings and propose three solutions that sharpen how we can measure physical understanding of VGMs. Specifically, we improve prompt and ground-truth quality to reduce the influence of confounding factors and further introduce a sample-level scoring system that weights each sample and metric equally. Our resulting benchmark, Physics-IQ Verified, refines 57.6\% of all samples and improves over 34.8\% of prompts. In a comparison study using six image-to-video generative models, we observe moderate but meaningful ranking changes (Kendall's τ= 0.46). We hope Physics-IQ Verified advances the community by providing a more reliable signal toward physically accurate VGMs. The code for the benchmark can be accessed at https://github.com/google-deepmind/physics-iq-benchmark