Physics-IQ Verified

Tim Rädsch

, Yuki M Asano

, Hilde Kuehne

, Stefan Bauer

Priyank Jaini

, Robert Geirhos

, Carsten T. Lüth

Anates Labs

Technical University of Munich

University of Technology Nuremberg

Tuebingen AI Center, University of Tuebingen

Helmholtz AI, Munich

research[at]anates[dot]ai

Abstract

\Acp

VGM have become a new frontier that can be used not just for video generation but for a multitude of downstream tasks, including world modeling. To advance these tasks, a good video model must understand the physical reality of the world. Evaluating this understanding is an emerging field and has led to the Physics-IQ benchmark [29], which quantifies this explicitly by comparing model-generated videos to real-world videos of physical experiments. In this work, we present a systematic audit of the Physics-IQ benchmark, expose shortcomings and propose three solutions that sharpen how we can measure physical understanding of video generative models. Specifically, we improve prompt and ground-truth quality to reduce the influence of confounding factors and further introduce a sample-level scoring system that weights each sample and metric equally. Our resulting benchmark, Physics-IQ Verified, refines 57.6% of all samples and improves over 34.8% of prompts. In a comparison study using six image-to-video generative models, we observe moderate but meaningful ranking changes (Kendall’s ). We hope Physics-IQ Verified advances the community by providing a more reliable signal toward physically accurate VGMs. The code for the benchmark can be accessed at Physiqs-IQ Verified Github.

Joint leads

1 Introduction

\Acp

VGM are increasingly positioned not merely as synthesis tools but as world models [33, 23, 10] which simulate the physical world for complex tasks in robotics [19] or as general visual task solvers [43]. This use is motivated by the assumption that the next-frame prediction objective implicitly teaches the model to encode the causal structure of physical reality [16]. This framing raises an immediate question:

How can we assess whether a model has actually learned to reason about the physical world, rather than learned to produce plausible-looking motion?

Earlier benchmarking efforts addressed this question using distributional metrics that compare unmatched sets of generated and real-world videos, such as Frechet Video Distance [38] or Frechet Video Motion Distance  [24]. The Physics-IQ Benchmark  [29] innovated this line of work by instead comparing model-generations to ground-truth recordings from controlled real-world physical experiments instead of simulated physics environments [40, 9, 37, 5, 32, 8, 47, 31, 20, 6, 2]. To quantify physical understanding it relies on four metrics that quantify where action occurs, when it occurs, how strongly it occurs, and how closely the generated frames match the ground truth at the pixel level. This design makes Physics-IQ one of the first benchmarks capable of directly measuring physical understanding rather than perceptual realism, and it has seen rapid adoption as a standard evaluation protocol for VGMs [48, 49, 35, 3, 52, 25, 27] also directly affecting model development  [48, 49, 35, 3, 25]. Therefore the fidelity of its scores to actual physical understanding becomes increasingly consequential.

We present an audit of Physics-IQ proposing three distinct improvements that reduce measurement errors arising from the evaluation protocol:

Improving Prompt quality. Some original prompts are ambiguous in their descriptions and prompting guidelines for models are not taken into account at all. We improve the quality of unclear text prompts by providing distinctive descriptions and by adhering to model-specific best practices for prompting using a templater. These two refinements ensure that the score reflects the capabilities of the evaluated model and minimizes the influence of suboptimal prompting.

Improving Metric Aggregation. The original Physics-IQ score is only defined on a dataset level, which also leads to samples having different influence on the final score. We define a sample-level Physics-IQ Verified score, which allows tracing back failure modes to each individual sample and weighs all samples and metrics equally.

Cleaning of Artifacts. Many videos contain “spurious metric activations” or artifacts that are not caused by the physical phenomena. We remove these artifacts from the ground truth of the reference videos, leading to the score more closely measuring the physical effect rather than unrelated and possibly random events.

Taken together, these contributions constitute Physics-IQ Verified: a refined benchmark that more faithfully reflects the ability of VGMs to model physical phenomena and allows a more fine-grained analysis of results by tracing back scores to a sample level. We provide an overview of our improvements in Figure 1 that highlights where the original benchmark is improved. The refinement removes possible measurement errors in 57.6% of all samples, influencing 29.8% of videos, correcting over 34.8% of prompts that are highly ambiguous (examples Figure 2 and detailed statistics Figure 3), while it also provides a template-based prompt structure with more accurate descriptions for all videos visualized in Figure 4. The benchmark is hosted at: https://github.com/google-deepmind/physics-iq-benchmark

Our evaluation of six image-to-video (I2V) VGMs using both the original and verified evaluation finds that models react differently to the improvements in evaluation, which leads to the overall ranking of models changing substantially. This highlights that VGM benchmarks must be carefully designed, so that models are tested on the effect of interest and to the best of their ability.

2 Background: The original Physics-IQ benchmark

The Physics-IQ benchmark contains 66 distinct physical experiments covering solid dynamics, fluid dynamics, thermodynamics, optics and magnetism. Each experiment is captured from three viewing angles and carried out twice resulting in overall videos (referred to as GT1 and GT2). These 8 second videos are then split into a 3 second conditioning part, and a 5 second “ground truth” video continuation for comparison. Each scenario includes an additional text description for conditioning. For the first 198 videos (ID001–198), switch frames mark the exact 3-second point where generation for the video generative model should begin. These switch frames, alongside previous video frames, can also be used as conditioning input for image-to-video or video-to-video models. The generation of VGMs is therefore constrained to 5 second videos on this first set of videos. The second set of videos (ID199–396) consists of second takes. These takes are used to compute the physical variation between identical setups. This variation serves as an upper performance ceiling, representing natural trial-to-trial variability.

Each video presents a physical experiment in which observable phenomena unfold after the switch-frame. The model’s task is to predict these phenomena based on a full prompt, whose composition depends on the model type: a text prompt alone for text-to-video (T2V) models, an image combined with text for I2V models, or a video clip or multiframe input combined with text for video-to-video (V2V) models.

Performance is measured using four metrics designed to quantify how closely the generated output replicates the physical phenomena. Three are activation-based[29, Algo. 2] Intersection over Union (IoU) metrics, and one is a pixel-based Mean Squared Error (MSE) metric: 1) Spatial IoU: Where does action happen? 2) Spatiotemporal IoU: Where & when does action happen? 3) Weighted spatial IoU: Where & how much does action happen? 4)  Mean Squared Error (MSE): How does action happen?

To compute the final Physics-IQ score, metric values are averaged and divided by the physical variation. This is followed by a weighted summation, with a negative sign applied to the MSE. The physical variation is obtained by computing the mean value for each of these metrics in the same way as for a normal evaluation but using the first and second take for each experiment. We give a detailed description of the used metrics with a clear mathematical notation in App. C.

Physics-IQ’s position among other benchmarks. Unlike judgment-based benchmarks that assess whether a video appears physically plausible [7, 28], or simulation benchmarks using synthetic data that test predefined physical rules [40, 9, 37, 5, 32, 8, 47, 31, 20, 6], Physics-IQ [29] compares generated continuations to real-world recordings of the same physical setup. This reference-based design makes Physics-IQ especially valuable because it provides a concrete physical target rather than a categorical plausibility judgment; at the same time, it makes the benchmark particularly sensitive to the quality of the ground-truth recordings. If prompts, reference activations, or aggregation choices include confounding factors, they directly change what physical effect is treated as the measurement target, motivating our audit. We provide a more detailed comparison to related benchmark families in App. F.

3 Physics-IQ Verified: Sharpening How Physical Understanding is Assessed

3.1 Improving text prompts

VGMs are steered through visual prompts, including conditioning frames of the initial state and text prompts describing the physical process. The prompt quality directly bounds what the benchmark can measure. Our proposed improvements address two sources of measurement error in the original benchmark: unclear prompts, and a lack of proper structure for specific models. We address each in turn, starting with a definition of a well designed prompt.

A well-designed prompt for assessing VGMs’ ability to model physics is a text description, accompanied by a conditioning frame or video, that clearly specifies the full experimental setup and the catalyst of the physical phenomenon, without revealing how that phenomenon unfolds.

The prompt should function as an exam question: a human provided with the prompt and start frame should be able to predict the experimental outcome with high confidence, yet the prompt must not make the answer obvious, lest it trivialize the generation task. Any ambiguity left unresolved by the prompt introduces degrees of freedom in the output that are orthogonal to physical understanding and therefore inflates metric variance or reduces performance irreducibly. Therefore, we depart from the original benchmark’s focus on scene description (Motamed et al. [29, p. 2]) in favor of clear experimental instructions.

We provide one concrete example with all the resulting changes detailed in the rest of this section in Figure 4.

3.1.1 Clarifying Unclear Prompts

Unclear prompts fail to narrow the space of plausible scenarios towards the specific scenario observed in the physical experiment. We identify four severity levels, ranging from making correct generation impossible to merely increasing output variance (see Figure 2 for examples). These are, in order of severity:

(1) Factually incorrect: does not match what happens in the video; (2) Temporally imprecise: fails to distinguish actions that have already occurred prior to the conditioning frame from following actions that should be generated; (3) Omitted key information: lacks information necessary to accurately model the physical effect; (4) Vague language: describes the observed action in terms that are too imprecise to sufficiently constrain the generation.

Factual incorrectness and temporal impreciseness make accurate generation impossible in principle. Omitted key information and vague language increase output variance by leaving physical degrees of freedom unconstrained. Each of these reduce the ability of both VGMs and humans to predict the physical effect reliably. This can bias the final score to reflect prompt clarity rather than model capability. Thus, here, we carefully screened and applied minimally invasive corrections yielding a complete set of updated descriptions.

3.1.2 Adhering to the Prompt–Model Interface

Independent of content quality, the original prompts are not structured according to the input conventions of the VGMs being evaluated. This manifests in the generation being insufficiently conditioned on the text prompt. Since the benchmark’s goal is to assess physical reasoning rather than robustness to naive user inputs, prompts should simulate an experienced user familiar with the target model.

To ensure consistent conditioning, we decompose each prompt into six structured fields. These six fields are used by model-specific templaters, which create the text prompt according to providers’ best practices (an example is shown in Figure 4). Three fields, namely SETUP, SCENE, and ACTION, capture variable, scenario-specific information adapted from the original prompts, while CAM, STYLE, and SCOPE remain consistent across all 66 scenarios. The latter two fields represent novel additions absent from the originals, each targeting a systematic gap.

STYLE constrains the rendering register to “…a realistic scientific demonstration”, preventing stylised or cartoonish outputs. SCOPE instructs “only contains the described setup and actions” to ensure the model is aware that no new actors or interactions enter the scene, suppressing hallucinated intrusions. The CAM field is changed to use descriptive cinematographic language describing the expected video in detail to ensure that it is sufficiently clear: “Static locked-off single-shot with fixed frame throughout, filmed at constant framerate in real-time.”. The importance of camera guidance is also evident in other works when evaluating VGMs [43].

A core principle while rewriting the prompts into our six fields is to express all instructions in positive terms as text-based negations are poorly handled by many models [36, 15, 30, 4, 12] and some model providers explicitly discourage them.¹¹1e.g. FLUX: https://docs.bfl.ml/guides/prompting_summary We provide more details with respect to this rewrite, the templater and the design process in App. A.2.

3.2 Improving Aggregation: Enforcing Equal Weights for each Sample and Metric

The original Physics-IQ score aggregates metrics across the entire dataset of samples as follows:

(1)

Here, is the metric value comparing the generated video to the reference (GT 1) for sample for the four metrics: spatial (SP)-, spatiotemporal (ST)-, weighted spatial (WS)-IoU and MSE. The clipping operation ensures the final score remains within . The physical variation acts as a normalization factor. It is obtained by comparing the second take of an experiment (GT 2) to the first take (GT 1), treating GT 2 as a baseline generation.

This dataset-wide aggregation has two structural issues, both stemming from the summation inside the denominator. First, the physical variation should reflect an upper bound for each specific experiment’s score. Averaging across the dataset invalidates this upper bound. Consequently, experiments with low physical variation are down weighted because they can never reach a score of 1. Conversely, experiments with high physical variation are up weighted, as their individual scores can exceed 1. Second, dataset-wide calculation obscures sample-level failures, making it difficult to trace low benchmark scores to specific failure modes. This reduces the benchmark’s utility for steering model development.

To solve these issues, we define the Physics-IQ Verified score directly at the sample level (). We aggregate the subscores using the arithmetic mean so that improvements in any metric are clearly reflected in the sample’s total score. To ensure the MSE is interpreted similarly to the IoU metrics (where higher is better), we define its influence as the inverse ratio of the MSE physical variation. This yields the following per-sample score:

(2)

The final Physics-IQ Verified score is the arithmetic mean across all samples: . Further details regarding the computation and the drawbacks of the original score are provided in App. C.

3.3 Cleaning of Spurious Metric Activations or Artifacts

All three IoU-based metrics used in Physics-IQ operate on activation maps[29, Algo. 2] that are derived from the visual differences of neighboring video frames, for both ground truth videos and generated videos. Because this applies to both ground truth and generated videos, the quality of the ground truth activations is crucial. High-quality activations ensure the metrics assess physical phenomena rather than “spurious activations” or artifacts. We, therefore, define:

An artifact as a metric activation caused by a visual event that is not part of the physical effect under observation. We distinguish them into two subtypes based on predictability:

• Deterministic artifacts stem from events that are specifiable from the prompt or experimental setup (e.g., a rotating apparatus). They are in principle predictable, but generate activation signal that is attributable to the apparatus rather than the physical phenomenon of interest.

• Non-deterministic artifacts arise by chance during recording and are absent from any prompt or experimental specification.

We show examples for both deterministic and non-deterministic artifacts alongside the result of our corrections in Figure 2 and provide their prevalence in Figure 3.

Both artifact types hinder assessment of physical understanding, but through distinct mechanisms. Deterministic artifacts bias the metric by adding activation signal that reflects apparatus behaviour rather than physical understanding, biasing scores in a structured way. Non-deterministic artifacts are more damaging from a measurement perspective. Because they are neither prompt-specified nor experimentally controlled, no model or human can anticipate them. This contributes entirely irreducible variance or bias to the benchmark scores.

We address both artifact types with a targeted removal strategy using manual annotations of the ground truth videos. First, we use end_effect_frames to indicate when the physical phenomenon ends, removing any artifacts that occur afterwards. Second, we use freeze_areas to pinpoint the spatial location and timing of artifacts occurring during the physical phenomenon. This allows us to remove artifacts that happen before the end_effect_frame. Details about artifact removal are provided in App. B.

4 Experiments

Experimental Setup.

We evaluate six I2V VGMs: three open-source, Wan 2.2 [39], HunyuanV-1.5 [22] and Cosmos3-N [1], and three closed-source, Sora 2 (v2025-10) [3], P-Video [14] and Grok Imagine Video [45]. We provide details with respect to all models in Table 2. Each model generates four complete sets of videos on the Physics-IQ dataset for both the original prompts (op) and our best-practice prompts (bpp), where each set consists of 198 videos following the standard I2V-protocol [29]. We perform evaluations in a factorial design that isolates the influence of each of our proposed evaluation improvements leading to 8 settings:  Prompt (op & bpp)  GT (original & verified)  score (original & verified). Detailed results are provided in App. E.

Method of Analysis.

The resulting rankings are analyzed using Kendall’s- [21] and Spearman’s- [34]; both metrics range from -1 to 1 and larger values indicate more agreement between rankings. Additionally we perform bootstrap analysis where 500 complete sets of videos of size 198 are generated by drawing for each video id the corresponding video from one of the four original sets. Based on this we estimate mean and 95% confidence intervals for Spearman’s- and Kendall’s-. We analyze absolute changes using Cohen’s d [11] to estimate the influence and Wilcoxon tests [44] to confirm statistical significance. During testing we evaluate each model run as an independent event following Demšar [13].

4.1 Comparing Original and Verified Evaluation

We compare the results of the original and our proposed verified evaluation using both the artifact removed ground truth and the best-practice prompts in Figures 5a&b. Overall, the scores increase for most models using Physics-IQ Verified compared to the original, mostly stemming from the improved prompts and our verified scores yielding higher values. Sora 2²²2The Sora 2 performance is notably worse in April 2026 than in October 2025. We confirm this in App. Tables 5&6. and Cosmos-3N have the highest increase in performance both outperforming P-Video. Overall, the verified evaluation produces a rank reshuffling: Grok Video and HunyuanV-1.5 move ahead of Wan 2.2 (the only model to reduce the score), Cosmos3-N and Sora 2 improve their positions, while P-Video falls from fourth to last place. The Spearman () and Kendall () correlations between rankings indicate moderate but meaningful changes. This is corroborated by the bootstrap analysis in Figure 5c: within-ranking correlations exceed , and their 95% confidence intervals do not overlap with the cross-evaluation ranking correlations (, ; see App. Figure 16 for details).

As these changes in scores and ranking are the result of three separate changes, we trace back the influence for each of these changes by assessing their impact on the original evaluation. We will start by giving the high-level takeaways and then discuss the details following this.

Overall better prompts improve the quality for all models with Sora 2 benefiting from it the most with Wan 2.2 being the only exception losing performance. Meanwhile artifact removal decreases the scores for all models but again most notably for Wan 2.2 indicating that some of its better score over other models stems from confounding effects. Changing the score from the original formulation to our sample level score yielded no change in overall ranking but increased the Physics-IQ score for all models.

4.2 Systematic Impact Assessment of each Improvement on the Original Evaluation

Influence of prompts.

Best-practice prompts (bpp) yield significantly significantly better sub-scores than original prompts (op) across all primary metrics in the original evaluation (Wilcoxon signed-rank: all ), with medium-to-large effect sizes (Cohen’s for all scores), as shown in Figure 6(a). The magnitude of improvement is model-dependent: for Sora 2, bpp prompts substantially reduce unwanted camera motion present under op prompts, driving large gains across all metrics. Wan 2.2 is the only model for which performance decreases under bpp, despite following guidelines for prompts.

Investigating influence of artifacts.

Removing evaluation artifacts significantly reduces performance across all IoU-based metric scores and the original Physics-IQ score (Wilcoxon signed-rank: all ), with large effect sizes (Cohen’s for all scores), as visualized in Figure 6(b). To identify the source of these reductions, we decompose score changes into numerator and denominator contributions. For most metrics, physical variance is nearly identical across protocols, so the score reduction is attributable entirely to the numerator. For the spatiotemporal metric, physical variance increases by under the verified protocol, introducing a denominator effect that mechanically suppresses scores independently of model behavior. These two mechanisms are structurally distinct and scores should not be compared across protocols without normalizing for this variance difference.

We hypothesize that the high degrees of freedom in the original prompts make it unlikely that any model interprets the intended physical scenario consistently, which would explain why bpp prompts reduce score variance in addition to improving mean performance; however, we did not observe this. We suspect that this might stem from the large degree of freedom in scenarios described by the original prompt making it very unlikely that the model is interpreting the prompt by chance close enough to its intended purpose to increase scores.

Influence or Benefit of Proposed Score on the Ranking.

In our evaluation our proposed score does yield higher Physics-IQ scores than the original formulation for all models. This almost uniform increase in scores does not change the ranking which is confirmed by the bootstrap ranking analysis resulting in almost perfect alignment values for both and . Details on both evaluations are provided in App. Figures 17&18. In our evaluation the main benefit of the proposed score therefore lies in the improved granularity which allows to trace back the influence of individual samples on the final score.

5 Conclusion

We presented a systematic audit of the influential Physics-IQ benchmark [29], whose finding that visual realism and physical understanding are largely uncorrelated has shaped subsequent work in the field. In our assessment, we identify three sources of measurement error and propose targeted solutions for each: text prompt improvements, artifact removal, and sample-wise score aggregation. Our experiments using six VGMs confirm that these changes impact the final evaluation in a significant way, with artifact removal reducing and improved prompts increasing absolute scores. Together, these refinements also change the final ranking of models. By providing the improved Physics-IQ Verified benchmark, we improve the measurement of physics of VGMs and hope to enable building the next generation of physically accurate VGMs.

Acknowledgments and Disclosure of Funding

The authors would especially like to thank Tassilo Wald for his detailed feedback on multiple drafts of this paper. We also thank Pruna AI for providing model credits to access their model. P.J. and R.G. contributed in an advisory capacity.

References

• [1] N. Agarwal, A. Ali, J. Allen, M. Antolini, A. Aubame, A. Azzolini, J. Bai, M. Bala, Y. Balaji, J. Bapst, et al. (2026) Cosmos 3: omnimodal world models for physical ai. arXiv preprint arXiv:2606.02800. Cited by: §4.
• [2] N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. (2025) Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. Cited by: §1.
• [3] O. AI (2025-09) Sora 2 system card openai september 30, 2025 1. External Links: Link Cited by: Appendix F, §1, §4.
• [4] K. Alhamoud, S. Alshammari, Y. Tian, G. Li, P. H. Torr, Y. Kim, and M. Ghassemi (2025) Vision-language models do not understand negation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29612–29622. Cited by: §A.3, §3.1.2.
• [5] T. Ates, M. Ateşoğlu, Ç. Yiğit, I. Kesen, M. Kobas, E. Erdem, A. Erdem, T. Goksun, and D. Yuret (2022) Craft: a benchmark for causal reasoning about forces and interactions. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2602–2627. Cited by: Appendix F, §1, §2.
• [6] A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick (2019) Phyre: a new benchmark for physical reasoning. Advances in Neural Information Processing Systems 32. Cited by: Appendix F, §1, §2.
• [7] H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y. Bitton, C. Jiang, Y. Sun, K. Chang, and A. Grover (2024) VideoPhy: evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520. Cited by: Appendix F, §2.
• [8] F. Baradel, N. Neverova, J. Mille, G. Mori, and C. Wolf (2019) Cophy: counterfactual learning of physical dynamics. arXiv preprint arXiv:1909.12000. Cited by: Appendix F, §1, §2.
• [9] D. M. Bear, E. Wang, D. Mrowca, F. J. Binder, H. F. Tung, R. Pramod, C. Holdaway, S. Tao, K. Smith, F. Sun, et al. (2021) Physion: evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.08261. Cited by: Appendix F, §1, §2.
• [10] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. (2024) Genie: generative interactive environments. In Forty-first International Conference on Machine Learning, Cited by: §1.
• [11] J. Cohen (1977) Statistical power analysis for the behavioral sciences, rev. Lawrence Erlbaum Associates, Inc. Cited by: §4.
• [12] C. Conwell, R. Tawiah-Quashie, and T. Ullman (2024) Relations, negations, and numbers: looking for logic in generative text-to-image models. arXiv preprint arXiv:2411.17066. Cited by: §A.3, §3.1.2.
• [13] J. Demšar (2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7 (Jan), pp. 1–30. Cited by: §4.
• [14] (2023) Efficient machine learning with pruna. Note: Software available from pruna.ai, Accessed: 2026-04-29 External Links: Link Cited by: §4.
• [15] I. García-Ferrero, B. Altuna, J. Alvez, I. Gonzalez-Dios, and G. Rigau (2023) This is not a dataset: a large negation benchmark to challenge large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 8596–8615. Cited by: §A.3, §3.1.2.
• [16] D. Ha and J. Schmidhuber (2018) World models. arXiv preprint arXiv:1803.10122 2 (3), pp. 440. Cited by: §1.
• [17] Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu (2024) VBench: comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix F.
• [18] Z. Huang, F. Zhang, X. Xu, Y. He, J. Yu, Z. Dong, Q. Ma, N. Chanpaisit, C. Si, Y. Jiang, Y. Wang, X. Chen, Y. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu (2024) VBench++: comprehensive and versatile benchmark suite for video generative models. arXiv preprint arXiv:2411.13503. Cited by: Appendix F.
• [19] J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y. Fang, F. Hu, S. Huang, K. Kundalia, Y. Lin, et al. (2025) DreamGen: unlocking generalization in robot learning through video world models. In Conference on Robot Learning, pp. 5170–5194. Cited by: §1.
• [20] B. Kang, Y. Yue, R. Lu, Z. Lin, Y. Zhao, K. Wang, G. Huang, and J. Feng (2024) How far is video generation from world model: a physical law perspective. arXiv preprint arXiv:2411.02385. Cited by: Appendix F, §1, §2.
• [21] M. G. Kendall (1945) The treatment of ties in ranking problems. Biometrika 33 (3), pp. 239–251. Cited by: §4.
• [22] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) Hunyuanvideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, Accessed: 2026-04-29. Cited by: §4.
• [23] Y. LeCun et al. (2022) A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62 (1), pp. 1–62. Cited by: §1.
• [24] J. Liu, Y. Qu, Q. Yan, X. Zeng, L. Wang, and R. Liao (2024) Fr’echet video motion distance: a metric for evaluating motion consistency in videos. arXiv preprint arXiv:2407.16124. Cited by: §1.
• [25] Y. Liu, X. Zhao, P. Wen, S. Dai, and Q. Huang (2025) Bootstrapping physics-grounded video generation through vlm-guided iterative self-refinement. arXiv preprint arXiv:2511.20280. Cited by: Appendix F, §1.
• [26] Y. Liu, X. Cun, X. Liu, X. Wang, Y. Zhang, H. Chen, Y. Liu, T. Zeng, R. Chan, and Y. Shan (2024) EvalCrafter: benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix F.
• [27] H. Lu, S. Wu, J. Zhang, M. Su, G. Ye, C. Xu, L. Lu, P. Maneriker, F. Du, M. Li, et al. (2026) Phys4D: fine-grained physics-consistent 4d modeling from video diffusion. arXiv preprint arXiv:2603.03485. Cited by: Appendix F, §1.
• [28] F. Meng, J. Liao, X. Tan, W. Shao, Q. Lu, K. Zhang, Y. Cheng, D. Li, Y. Qiao, and P. Luo (2024) Towards world simulator: crafting physical commonsense-based benchmark for video generation. arXiv preprint arXiv:2410.05363. Cited by: Appendix F, §2.
• [29] S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos (2026) Do generative video models understand physical principles?. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 948–958. Cited by: §A.4, §C.2, §C.5, Appendix F, §1, §2, §2, §3.1, §3.3, §4, §5.
• [30] L. Parcalabescu, M. Cafagna, L. Muradjan, A. Frank, I. Calixto, and A. Gatt (2022) VALSE: a task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8253–8280. Cited by: §A.3, §3.1.2.
• [31] N. F. Rajani, R. Zhang, Y. C. Tan, S. Zheng, J. Weiss, A. Vyas, A. Gupta, C. Xiong, R. Socher, and D. Radev (2020) ESPRIT: explaining solutions to physical reasoning tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7906–7917. Cited by: Appendix F, §1, §2.
• [32] R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux (2018) Intphys: a framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. Cited by: Appendix F, §1, §2.
• [33] J. Schmidhuber (1990) Making the world differentiable: on using self supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. Vol. 126, Inst. für Informatik. Cited by: §1.
• [34] C. Spearman (1961) The proof and measurement of association between two things.. Cited by: §4.
• [35] H. Teng, H. Jia, L. Sun, L. Li, M. Li, M. Tang, S. Han, T. Zhang, W. Zhang, W. Luo, et al. (2025) Magi-1: autoregressive video generation at scale. arXiv preprint arXiv:2505.13211. Cited by: Appendix F, §1.
• [36] T. H. Truong, T. Baldwin, K. Verspoor, and T. Cohn (2023) Language models are not naysayers: an analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (* SEM 2023), pp. 101–114. Cited by: §A.3, §3.1.2.
• [37] H. Tung, M. Ding, Z. Chen, D. Bear, C. Gan, J. Tenenbaum, D. Yamins, J. Fan, and K. Smith (2023) Physion++: evaluating physical scene understanding that requires online inference of different physical properties. Advances in Neural Information Processing Systems 36, pp. 67048–67068. Cited by: Appendix F, §1, §2.
• [38] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly (2019) FVD: a new metric for video generation. External Links: Link Cited by: Appendix F, §1.
• [39] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, Accessed: 2026-04-29. Cited by: §4.
• [40] M. Wang, R. Wang, J. Lin, R. Ji, T. Wiedemer, Q. Gao, D. Luo, Y. Qian, L. Huang, Z. Hong, et al. (2026) A very big video reasoning suite. arXiv preprint arXiv:2602.20159. Cited by: Appendix F, §1, §2.
• [41] Z. Wang, S. Li, L. Hao, X. Hu, and B. Song (2024) What you see is what matters: a novel visual and physics-based metric for evaluating video generation quality. arXiv preprint arXiv:2411.13609. Cited by: Appendix F.
• [42] D. M. Wegner, D. J. Schneider, S. R. Carter, and T. L. White (1987) Paradoxical effects of thought suppression.. Journal of personality and social psychology 53 (1), pp. 5. Cited by: §A.3.
• [43] T. Wiedemer, Y. Li, P. Vicol, S. S. Gu, N. Matarese, K. Swersky, B. Kim, P. Jaini, and R. Geirhos (2025) Video models are zero-shot learners and reasoners. arXiv preprint arXiv:2509.20328. Cited by: §A.4, §1, §3.1.2.
• [44] F. Wilcoxon (1945) Individual comparisons by ranking methods. Biometrics bulletin 1 (6), pp. 80–83. Cited by: §4.
• [45] xAI (2026) Grok Imagine API: state-of-the-art video generation across quality, cost, and latency. Note: https://x.ai/news/grok-imagine-apiAccessed: 2026-04-29 Cited by: §4.
• [46] Q. Xue, X. Yin, B. Yang, and W. Gao (2025) Phyt2v: llm-guided iterative self-refinement for physics-grounded text-to-video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18826–18836. Cited by: §D.1.
• [47] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum (2019) Clevrer: collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442. Cited by: Appendix F, §1, §2.
• [48] J. Yuan, X. Zhang, F. Friedrich, N. Beltran-Velez, M. Hall, R. Askari-Hemmat, X. Han, N. Ballas, M. Drozdzal, and A. Romero-Soriano (2025) Improving the physics of video generation with vjepa-2 reward signal. arXiv preprint arXiv:2510.21840. Cited by: Appendix F, §1.
• [49] J. Yuan, X. Zhang, F. Friedrich, N. Beltran-Velez, M. Hall, R. Askari-Hemmat, X. Han, N. Ballas, M. Drozdzal, and A. Romero-Soriano (2026) Inference-time physics alignment of video generative models with latent world models. arXiv preprint arXiv:2601.10553. Cited by: Appendix F, §1, footnote 4.
• [50] C. Zhang, D. Cherniavskii, A. Tragoudaras, A. Vozikis, T. Nijdam, D. W. Prinzhorn, M. Bodracska, N. Sebe, A. Zadaianchuk, and E. Gavves (2025) Morpheus: benchmarking physical reasoning of video generative models with real physical experiments. arXiv preprint arXiv:2504.02918. Cited by: Appendix F.
• [51] D. Zheng, Z. Huang, H. Liu, K. Zou, Y. He, F. Zhang, Y. Zhang, J. He, W. Zheng, Y. Qiao, and Z. Liu (2025) VBench-2.0: advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755. Cited by: Appendix F.
• [52] S. Zhuang, Z. Huang, Y. Zhang, F. Wang, C. Fu, B. Yang, C. Sun, C. Li, and Y. Wang (2025) Video-gpt via next clip diffusion. arXiv preprint arXiv:2505.12489. Cited by: Appendix F, §1.

Appendix

Appendix A Prompt Improvements

A.1 Qualitative Prompt Examples

A.2 Prompt Template Design

A well-designed prompt should function as an exam question: a human given the prompt and start frame should be able to predict the experimental outcome with high confidence, but the prompt must not make the answer obvious, lest it trivialise the generation task. Any ambiguity left unresolved by the prompt introduces degrees of freedom in the output that are orthogonal to physical understanding and therefore inflate metric variance or reduces performance irreducibly.

媒体内容 · 前往原文查看

Symbol	Type	Content
SETUP	Variable	Pre-action scene description: objects, their spatial arrangement, and initial conditions prior to any physical event.
SCENE	Variable	Scene description: supplementing SETUP with temporally constant information.
ACTION	Variable	Subject-action description.
CAM	Fixed	Camera and recording specification; enforces a static, locked-off, constant-framerate shot.
STYLE∗	Fixed	Rendering register; constrains output to a realistic scientific demonstration.
SCOPE∗	Fixed	Content boundary; instructs the model that no new actions take place during this video.

A.3 Avoiding Negations

A core principle of the rewrite is to express all instructions in positive terms, motivated on three independent grounds. From the model perspective, it is a known phenomenon that text-based negations are poorly handled which likely extends to video-models given that it has been observed for LLMs [36, 15], vision–language models such as CLIP [30, 4], and text-to-image generative models [12] that they all exhibit systematic failures with negated instructions. For the human psyche, suppressing a concept reliably activates it, a phenomenon formalised as ironic process theory by Wegner et al. [42]. Finally, positive framing is explicitly recommended in provider prompting guidelines.³³3e.g. FLUX: https://docs.bfl.ml/guides/prompting_summary

A.4 Camera Guidance

Cinematographic consistency is particularly consequential for this benchmark: evaluation metrics penalise deviations in camera pose and motion between generated and ground-truth video. The original prompts specify only “Static shot with no camera movement”. Motamed et al. [29] themselves acknowledge that given this setup many models and especially Sora are still subject to camera drift. The importance of more thorough cinematographic specification becomes clear implicitly when reading instructions like “Static camera perspective, no zoom no pan no movement no dolly no rotation” in Wiedemer et al. [43, Figs. 10–26]. Applying the positive-framing principle consistently, we formalise these findings into a single fixed CAM field: “Static locked-off single-shot with fixed frame throughout, filmed at constant framerate in real-time.” This replaces negation-based instructions with descriptive cinematographic language and is applied uniformly across all scenarios.

Appendix B Artifact Cleaning and Dataset Modification

We address each artifact type through a targeted removal strategy. Both strategies rely on manual annotation of artifact extent, encoded as {annotation} in the dataset, and employ frame freezing as the removal primitive which corresponds to holding pixel values constant in the affected region from a given timestamp onward. Freezing is preferred over alternatives such as masking or inpainting because it introduces no new visual information and avoids artificial boundaries that could themselves generate spurious metric activations.

• Post-effect removal targets artifacts occurring after the physical effect has concluded. Frames are frozen beyond a manually annotated endpoint specified via {end_effect_frames}, eliminating all post-effect visual events regardless of their spatial location. This primarily addresses Non-deterministic artifacts in the temporal tail of the video.

• Mid-effect removal targets artifacts occurring during the physical effect in regions that are spatially disjoint from it. Designated spatial regions are frozen from a manually annotated timestamp onward, specified via {freeze_areas}. This strategy handles both deterministic apparatus artifacts and incidental non-deterministic events that overlap temporally with the effect.

We provide visual examples for representative artifact corrections and a dataset-wide overview of our applied changes in Figure 13.

B.1 Qualitative Artifact Examples

B.2 Dataset-Wide Modification Overview

Appendix C Detailed Metric Definition

C.1 Key improvements from the original to the verified Physics-IQ evaluation.

C.2 Variables and Derived Maps

The dataset consists of videos, where experiments are each captured with takes across viewing angles. Each video is a tensor , where denotes spatial resolution and the number of frames.

From each video, a binary spatiotemporal activation map is derived from the greyscale signal, encoding where and when motion or activation occurs (see [29, Algo. 2] for details). Two further representations are derived from this map for use in the metrics:

• The spatial activation map captures where any activation occurred across the full video.

• The weighted spatial activation map captures where and how much activation occurred, weighted by temporal frequency. Note that whether or not normalization is applied does not affect the resulting Weighted-Spatial-IoU score.

C.3 Basic Metric Definitions

Three IoU-based metrics are defined over the activation maps, and one pixel-level reconstruction metric over the raw video:

	Spatial-IoU			(3)
	Spatiotemporal-IoU			(4)
	Weighted-Spatial-IoU			(5)
				(6)

where the MSE is computed as the mean over all frames of an experiment, with videos normalised to .

The metric values for a single sample and the corresponding physical variation are defined as:

				(7)
				(8)
				(9)
				(10)

C.4 Original Physics-IQ Score Aggregation

Each metric is aggregated over the evaluation videos into a mean score and a physical variation ceiling . The ceiling is computed by comparing the two ground-truth takes of each experiment, quantifying the irreducible trial-to-trial variability of the physical phenomena.

For the IoU metrics the subscores are defined as:

				(11)
				(12)
				(13)

For MSE, lower is better, so the ceiling is subtracted rather than used as a divisor:

				(14)
				(15)
				(16)

C.5 Stable Physics-IQ Score

For the original composite Physics-IQ score the three IoU sub-scores are averaged and the MSE penalty is subtracted to produce a raw composite score, which is then clipped to :

(17)

where each subscore is normalised by the physical variation ceiling, representing the typical deviation between independent second takes of the same experiment. The structural flaw in Eq. 17 is that the scores for each metric, Spatial, Spatiotemporal and weighted spatial IoU (), are unbounded: a single exceptional subscore can dominate the composite irrespective of performance on the remaining metrics, directly contradicting the design intent of Motamed et al. [29] that “no metric should be assessed in isolation.” By construction, a subscore of for the positive metrics indicates that the generated videos match the ground truth as well as a second take would; scores above indicate that estimated ceiling performance has been surpassed, which the outer does not prevent from inflating the average before aggregation.

As Physics-IQ is designed to assess physical understanding relative to natural scene variability, not to reward performance beyond second-take realism. We therefore enforce a performance ceiling at the physical variation by clipping each subscore individually before aggregation resulting in the Physics-IQ stable composite score:

(18)

The symmetry here is one of design intent rather than mathematical range. For :  indicate better-than-ceiling performance and are clipped to . For :  indicate better-than-ceiling pixel similarity and are clipped to . The symmetry here is one of design intent rather than mathematical range. Per-metric clipping ensures that no individual subscore can contribute beyond its intended share of the composite, while preserving full sensitivity in the practically relevant regime of below-ceiling performance. This correction is principled regardless of empirical impact; where it additionally affects model rankings, this reflects the degree to which the original formula was distorted by subscore dominance.⁴⁴4At the time of writing the highest scores for the original Physics-IQ benchmark is at [49]. Therefore video generative models are not close yet in any metric towards hitting the performance ceiling.

C.6 Drawbacks of the Original Score

• The score is not defined for a single sample but only over the entire dataset. This makes it unclear on which samples a model performs well and on which samples it does not perform well.

• The mean aggregation for the physical variation leads to smaller values contributing less to the overall score. Samples that have a smaller physical variation and in theory also smaller scores contribute less to the overall score.

• The original unclipped score could have an overflow for sub-score values greater than 1 (or smaller than 0 for MSE). A single very high score can dominate the Physics-IQ score.

C.7 Sample-Level Physics-IQ Verified Score

We propose a principled physics-iq score operating on the sample level over the entire dataset. The aggregation of each score is performed using the arithmetic mean so that improvements across every single metric are clearly attributed in the per sample score.

Additionally, we change the interpretation of the MSE which now captures how many times the generated MSE score is larger than that of the physical variation.

(19)

The final Physics-IQ Verified score is the arithmetic mean across all samples:
.

The subscores for each metric over the entire dataset for our verified scores are obtained by summing over all samples in an identical fashion.

Appendix D Experimental Setup

D.1 Evaluated Models

We provide details with respect to our evaluated models in Table 2.

媒体内容 · 前往原文查看

Model	Text	v2v	i2v	Size	FPS	Resolution	Seed Control	Price
Grok Imagine Video				n.d.	24	1280720		$0.352
HunyuanV-1.5				8.3B	24	848480		$0.400
P-Video				n.d.	24	1280704		$0.100
Sora-2				n.d.	30	1280720		$0.800
Wan 2.2				14B	16	1280720		$0.110
Cosmos3-Nano				16B	24	1280720		$0.333

For Cosmos3-Nano, we hand both op and bpp prompts directly to the VGMs without preprocessing them using a LLM or VLM. This decision is motivated by our aim to accurately capture the influence of the prompts, additionally the official i2v leaderboard score of Cosmos3 makes use of prompts generated using PhyT2V [46] which does not adhere to their proposed prompting structure⁵⁵5recommendation for upsampling Cosmos3 Nano, uploaded i2v prompts for official submission. Cosmos3-Nano bpp with Opus 4.8 upsampling improves over Cosmos3-Nano bpp without upsampling by ~1 Physics-IQ verified score point in our separate experiments.

Appendix E Additional Results

This section reports the full quantitative results underlying Section 4. We present results for both the Physics-IQ Original Score and the Physics-IQ Verified Score in Table 3 and 4.

Table 5 and 6 provide additional Sora 2 sanity checks across evaluation dates and generation settings.

Figure 16, 17, and 18 show the bootstrap ranking analysis used to assess ranking stability.

E.1 Main Results Overview

媒体内容 · 前往原文查看

Phys-IQ orig. SP orig. ST orig. WS orig. MSE orig. Model Ground Truth Prompt Cosmos3-N original bpp op verified bpp op Grok Video original bpp op verified bpp op HunyuanV-1.5 original bpp op verified bpp op P-Video original bpp op verified bpp op Sora 2 original bpp op verified bpp op Wan2.2 original bpp op verified bpp op

媒体内容 · 前往原文查看

Phys-IQ Verified SP verified ST verified WS verified MSE verified Model Ground Truth Prompt Cosmos3-N original bpp op verified bpp op Grok Video original bpp op verified bpp op HunyuanV-1.5 original bpp op verified bpp op P-Video original bpp op verified bpp op Sora 2 original bpp op verified bpp op Wan2.2 original bpp op verified bpp op

E.2 Sora-2 Temporal Comparison

媒体内容 · 前往原文查看

Phys-IQ orig. SP orig. ST orig. weighted SP orig. MSE orig. Model Ground Truth Prompt Sora 2 original bpp op verified bpp op Sora 2 (12-25) original bpp op verified bpp op Sora 2 (10-25) original op verified op

媒体内容 · 前往原文查看

Phys-IQ Verified SP verified ST verified WS verified MSE verified Model Ground Truth Prompt Sora 2 original bpp op verified bpp op Sora 2 (12-25) original bpp op verified bpp op Sora 2 (10-25) original op verified op

E.3 Bootstrap Ranking Analysis

Appendix F Related Works

Video generation evaluation beyond perceptual realism.

Early evaluation of video generative models (VGMs) largely focused on perceptual quality, distributional similarity, and semantic alignment, using metrics such as Fréchet Video Distance (FVD) [38] or broad evaluation suites. More recent benchmarks decompose video quality into more fine-grained axes: VBench [17] and VBench++ [18] evaluate dimensions such as motion smoothness, temporal flickering, spatial consistency, subject identity, and prompt alignment, while EvalCrafter [26] assesses visual, content, and motion quality across a diverse prompt set. VBench-2.0 [51] further extends this line toward intrinsic faithfulness, including dimensions related to commonsense and physical plausibility. These benchmarks are important for measuring whether videos are visually coherent and semantically aligned, but they do not directly test whether a generated continuation follows the causal physical dynamics of a real experiment. This distinction motivates a separate line of work on physical understanding in VGMs.

Synthetic and simulator-based physical reasoning benchmarks.

Before the recent focus on VGMs, physical reasoning was often studied in synthetic or simulated environments. PHYRE [6] introduced a 2D physical reasoning benchmark in which agents solve classical mechanics puzzles by interacting with a simulated world. Physion [9] evaluates whether models can predict the future evolution of physical scenes, while Physion++ [37] extends this setting to scenarios requiring online inference of latent physical properties. Other synthetic benchmarks, including IntPhys [32], CoPhy [8], CLEVRER [47], CRAFT [5], and ESPRIT [31], similarly test intuitive physics, causal reasoning, or counterfactual prediction under controlled conditions. These benchmarks provide strong experimental control and often allow large scale testing. The largest benchmark to date comprises over 10 million synthetic clips generated from 200 curated tasks, a large share of which target physical reasoning, while others cover non-physical reasoning tasks such as Sudoku [40]. However, they differ from the current VGM setting because the data are typically rendered or simulated rather than recorded from real-world camera videos. Thus, they do not fully capture the visual ambiguity, apparatus effects, lighting conditions, and recording artifacts that arise when evaluating modern video generators on real physical experiments, and ultimately define the sim-to-real gap.

Physical reasoning benchmarks for video generative models.

Recent work has adapted physical reasoning evaluation to the VGM setting. One family relies on human or vision-language-model judgments. VideoPhy [7] evaluates whether generated videos obey physical commonsense in everyday material interactions, while PhyGenBench [28] curates prompts covering multiple physical laws and uses a hierarchical evaluation protocol. These benchmarks are scalable and cover many physical concepts, but their judgments are primarily categorical: they can identify that a generation violates a physical expectation, but they do not necessarily quantify how strongly or where the violation occurs.

A second family uses motion-, mask-, or trajectory-based proxies. VAMP [41] proposes visual appearance and motion-plausibility metrics based on quantities such as acceleration and velocity variance. Kang et al. [20] evaluate video generation from a physical-law perspective in synthetic environments, studying whether scaling improves the ability of VGMs to model classical mechanics. These approaches move beyond pure perceptual realism, but they either remain tied to synthetic environments or use proxy motion statistics rather than real-world reference experiments.

The third family grounds evaluation in controlled physical settings: Morpheus [50] introduces physics-informed neural networks (PINNs) to assess whether generated trajectories conform to governing equations and conserved physical invariants, such as total energy and angular momentum, derived from real laboratory experiments. This provides a complementary law-based perspective to Physics-IQ. However, Morpheus is limited to object-centric phenomena for which reliable trajectories can be extracted and for which the relevant dynamics can be expressed through low-dimensional state variables. As a result, many physical effects covered by Physics-IQ, such as drops falling into water, fluid motion, splashes, diffuse material changes, or phenomena where the relevant signal is not a single object trajectory, are not naturally captured by Morpheus-style trajectory metrics.

Physics-IQ.

Physics-IQ [29] is the most direct predecessor of our work. It introduces a dataset of 396 real-world videos spanning five physical domains: fluid dynamics, optics, solid mechanics, magnetism, and thermodynamics. The benchmark adopts a prediction-from-context paradigm: models are conditioned on a starting image or video clip and must generate the physical continuation of the scene. Physical understanding is evaluated through pixel-level comparison between generated and ground-truth continuations using Spatial IoU, Spatiotemporal IoU, and Mean Squared Error (MSE), aggregated into a composite Physics-IQ score normalized by the natural variance observed across real-world reference videos. Evaluations of Sora, Runway, Pika, Lumiere, Stable Video Diffusion, and VideoPoet reveal that physical understanding is severely limited across all tested models, and that visual realism is largely independent of physical accuracy [29] . Physics-IQ established an important empirical foundation and a reproducible evaluation pipeline. Nonetheless, several limitations remain. Mask-based overlap metrics are bounded by the quality of segmentation and may conflate spatial proximity with physical correctness. They also presuppose that the reference trajectory represents the uniquely correct physical outcome, which can yield false negatives when a generated video is physically plausible but explores a different, yet valid, realization of the scene. The benchmark does not assess whether conserved quantities such as energy or momentum are preserved in generated sequences, and the set of models evaluated has been substantially superseded by newer architectures.

Adoption of Physics-IQ as an benchmark and development target.

Physics-IQ has rapidly become more than a standalone benchmark. It has been used as an evaluation protocol for recent video generation systems and physics-aware model development. MAGI-1 [35] reports Physics-IQ results to assess physical continuation quality in autoregressive video generation. Yuan et al. [48, 49] use Physics-IQ to evaluate whether VJEPA-2-based reward signals and inference-time alignment can improve the physical plausibility of generated videos. The ICCV 2025 Physics-IQ Challenge further institutionalized the benchmark as a shared evaluation target, with follow-up methods such as VLM-guided iterative self-refinement [25] directly optimizing performance on the Physics-IQ task. Additional recent model papers, including Sora 2[3], also report Physics-IQ scores when claiming improvements in physically consistent video generation [35, 3, 52, 25, 27].

This adoption strengthens the motivation for our audit. Once a benchmark becomes a standard reporting protocol and an optimization target, measurement errors can propagate into model-development decisions. Prompt ambiguities, spurious ground-truth activations, and aggregation artifacts no longer only affect one benchmark paper; they can shape which systems appear more physically capable and which design choices are rewarded. Physics-IQ Verified addresses this issue by preserving the real-world continuation setting of Physics-IQ while improving prompt quality, cleaning artifact-driven activations, and introducing a sample-level aggregation scheme that makes benchmark outcomes more traceable and reliable.