# StatePlay：面向机制一致游戏生成的状态感知世界模型 - 来源：HuggingFace Daily Papers（社区热门论文） - 发布时间：2026-07-29 08:00 - AIHOT 分数：59 - AIHOT 链接：https://aihot.virxact.com/items/cms7c4uuk08plro2eoa2nds8d - 原文链接：https://arxiv.org/abs/2607.26754 ## AI 摘要 腾讯研究团队提出 StatePlay，一种状态感知游戏世界模型，能联合预测游戏帧与内部状态（血量、技能槽、计时器），确保生成内容符合游戏规则。StatePlay 采用混合 Transformer（MoT）架构，分别优化视觉与状态分支并实现跨模态交互。实验显示，其状态预测平均归一化 L1 距离低于 0.06，生成游戏回放的机制保真度相比无显式状态建模的模型提升 18.6%。 ## 正文 Abstract Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation. Project Page: https://jimntu.github.io/stateplay_page/ This work was completed during research internships at Tencent under the supervision of Yeying Jin. $\ddagger$ $\ddagger$ 1 Introduction Figure 1: Existing game world models produce mechanically inconsistent gameplay, such as attacking at zero health or triggering a super art without a full skill meter (red boxes, top). In contrast, StatePlay jointly achieves state awareness and mechanics consistency (bottom table). World models have attracted increasing attention for simulating future environment dynamics conditioned on agent or user actions (Bruce et al. 2024; Mao et al. 2026). Unlike traditional video generation models (Seedance et al. 2026; Team et al. 2025), they emphasize action-conditioned prediction, where future observations are jointly determined by historical context and interactive inputs. This capability makes them well suited to closed-loop applications such as autonomous driving (Wang et al. 2024), embodied AI (Ye et al. 2026), and games (Yu et al. 2025). In particular, games provide a promising testbed because the environment must respond coherently to player actions over time. Recent game world models can generate visually realistic and controllable gameplay rollouts while supporting interactions with dynamic environments (Tong et al. 2026), non-player characters (Wang et al. 2026a), and multiple players (Wu et al. 2026), enabling more immersive, creative, and engaging gameplay experiences. Figure 2: Overview of StatePlay. By explicitly modeling game states, including timers, health points, and skill meters, StatePlay guides frame generation to remain consistent with the underlying game mechanics (highlighted in green bounding box). However, simulating a playable game requires more than generating realistic frames that follow user intent. Games are governed by underlying mechanics, which are often determined by internal state variables such as health points, skill meters, and timers. These states regulate valid actions, visual transitions, and game progression. Taking fighting games as an example, a player can perform certain special attacks only when the skill meter is full, and the game should end once either player’s health points reach zero. Therefore, valid and consistent mechanics are fundamental to games, directly determining whether a generated environment is truly playable or merely a visual demo. Despite the importance of internal states, existing game world models formulate gameplay generation as a pixel-space prediction task and overlook the causal role of state dynamics in shaping future observations. Consequently, current models may generate visually plausible rollouts but fail to follow the game rules. As shown in Fig. 1, players may continue fighting after the game should have ended, or perform super arts without full skill meters. Such failures lead to mechanically inconsistent and incomplete gameplay experiences. To address this gap, we introduce StatePlay, the first state-aware game world model that jointly predicts game states and visual content for mechanics-consistent gameplay generation (See Fig. 2). Our key observation is that player actions are often directly reflected in pixel-level changes, whereas internal state dynamics cannot be reliably learned from visual observations alone. Therefore, existing game world model datasets that contain only frames and actions are insufficient for learning state transitions and game rules. Based on this observation, we develop StatePlay from both the data and model perspectives. First, due to the limited availability of open-source datasets with explicit state annotations, we construct a synchronized state–frame–action dataset from Street Fighter 3 (SF3) (Capcom 1998). The dataset records game frames and player actions together with internal states, including health points, skill meters, and timers, enabling us to systematically investigate the effect of explicit state modeling on game world models. Second, inspired by the World Action Model (WAM) (Ye et al. 2026; Kim et al. 2026), which jointly models visual frames and actions, we design StatePlay with a mixture-of-transformers (MoT)-style backbone that simultaneously predicts game states and generates visual content. Specifically, StatePlay couples the two modalities while optimizing each branch with a tailored objective to produce rollouts that are both visually plausible and consistent with game mechanics. Experiments highlight the importance of generating visual content with explicit state prediction. StatePlay accurately models internal game states, achieving an average normalized L1 distance below 0.06 across key state variables. Beyond state prediction itself, coupling states with visual generation leads to a 18.6% improvement in mechanics fidelity over the best-performing baseline. Compared with stateless game world models that learn game mechanics implicitly through pixel-level supervision, StatePlay better preserves state-dependent events, such as skill activation, health reduction, and game termination, while maintaining visual quality and action controllability. These results indicate that explicit state modeling is crucial for advancing game world models beyond visual realism toward more playable and mechanically consistent simulation. Overall, our contributions are summarized as follows: • We propose StatePlay, the first state-aware game world model that jointly predicts game states and visual content, addressing the lack of state modeling in prior work. • We introduce a novel mixture-of-transformers (MoT)-style architecture with distinct objectives for the state and visual branches. By explicitly coupling state modeling with visual generation, it enables accurate state prediction and mechanically consistent frame generation. • We conduct extensive experiments demonstrating that StatePlay improves mechanics fidelity by 18.6% over stateless game world models, while maintaining accurate state prediction, action controllability, and visual quality. 2 Related Work 2.1 Game World Model Game world models have the potential to transform game development by autonomously generating interactive content conditioned on player actions. Pioneering work such as Genie (Bruce et al. 2024) demonstrates that video generative models can produce playable environments by learning action-controllable dynamics from unlabeled videos. Studies such as ReactiveGWM (Wang et al. 2026a), Incantation (Zhu et al. 2026), and MultiWorld (Wu et al. 2026) further extend this direction toward richer forms of interactivity, from free exploration to interactions with environments, NPCs, and other players. In parallel, recent works including Lyra 2.0 (Shen et al. 2026), HY-World 1.5 (Sun et al. 2025), and Matrix-Game 3.0 (Wang et al. 2026b) focus on improving real-time interaction, long-horizon consistency, and high-resolution generation. Despite rapid progress in game world models, state prediction, which is equally important for building playable environments, remains largely overlooked. In this work, we identify this missing component in recent game world model research and enable game world models to simultaneously predict game states and generate video frames that follow the underlying game rules. 2.2 State-Aware World Modeling The use of state information has been widely studied in embodied and decision-making domains, where the definition of state depends on the task setting. In embodied AI, state often refers to proprioceptive information, such as robot poses and joint configurations, which helps policies generate physically feasible actions (Black et al. 2025; Kim et al. 2024). In autonomous driving, state variables such as ego speed, acceleration, and surrounding object motion are commonly modeled to support safer future prediction and planning (Zheng et al. 2024; Hu et al. 2022). These examples suggest that explicit state modeling is crucial for systems that require both visual understanding and rule-consistent interaction. Games also involve task-dependent states, such as ammunition or bandage counts in first-person shooter (FPS) games, nitro or speed in racing games, and health points or skill meters in fighting games. However, this perspective remains underexplored in game world models because game state information cannot always be directly extracted from the environment or reliably inferred from generated content. The most closely related recent works, including WildWorld (Li et al. 2026a) and From Pixels to States (Li et al. 2026b), introduce action-conditioned video datasets with explicit state annotations, but it remains unclear how state prediction can improve the mechanics fidelity of the frame generation, or how such state information should be incorporated into game world models. In this work, we propose StatePlay, a state-aware game world model, to fill this gap and systematically investigate how visual generation and state prediction can complement and reinforce each other. 3 StatePlay 3.1 State-Aware Dataset Construction We specifically construct our dataset using Street Fighter 3 (SF3) (Capcom 1998), as few publicly available game datasets provide explicit state annotations. This game offers a suitable starting point for studying the relationship between state prediction and mechanics-consistent visual generation, as it contains intuitive state variables and clearly defined game rules. These states are directly tied to game progression: different attacks cause different amounts of health-point reduction, super arts can be activated only when the skill meter is full, and the game ends when either character’s health reaches zero. The state-aware dataset is constructed in three stages: gameplay recording, mechanics distribution balancing, and NPC strategy annotation. Gameplay Recording Following a similar data acquisition pipeline to ReactiveGWM (Wang et al. 2026a), the stable-retro framework (Poliquin 2026) is employed to programmatically collect gameplay episodes. It provides a Gymnasium-compatible interface for SF3, allowing us to control the game through Python, record visual observations and actions. Players are controlled by agents that sample actions from an 11-dimensional action space, including 4 movement actions, 6 normal attacks, and 1 super art. Additionally, we extract five state variables from emulator memory: the game timer, the health points and the skill meters of both the player and the NPC. Each episode runs until a round-ending knock-out, and is then segmented into 5-second clips at 20 FPS. Mechanics Distribution Balancing After collecting the gameplay clips with synchronized player actions and internal states, we balance the dataset across five mechanics-related categories: result win, result lose, macro success, macro fail, and normal. Labels are assigned using both Gemini-3.1-Pro (Google DeepMind 2026) and the recorded game states. Specifically, a clip is labeled as result win or result lose when Gemini detects the corresponding “You Win” or “You Lose” frame, and one character reaches zero health. A clip is labeled as macro success when Gemini detects a successfully executed super art, the recorded action contains the corresponding command, and the skill meter exceeds the required threshold. In contrast, it is labeled as macro fail when the player issues the command with insufficient skill meter, causing a regular attack instead. The remaining clips without these state-dependent events are labeled as normal. Finally, we construct a balanced training set with 10,000 clips. State-critical cases account for 40% of the dataset, including result win, result lose, macro success, and macro fail, each contributing 10%. The remaining 60% consists of normal cases. This balancing strategy ensures that the dataset contains sufficient clips where state information is essential for generating mechanics-consistent game frames, while still preserving common gameplay dynamics. Please refer to Appendix B.1 for more details. Figure 3: StatePlay block architecture during training. StatePlay adopts a Mixture-of-Transformers (MoT)-style architecture with separate state and visual branches. A shared joint-attention module enables information exchange between the two branches. The state branch is optimized with a regression loss, while the visual branch is trained using a flow-matching objective. White-filled boxes denote inputs and supervision targets, while colored boxes represent model components. The block is repeated times. NPC Strategy Annotation To enable the model to steer the NPC toward different strategies against the player, we use Gemini-3.1-Pro to annotate NPC behavior with natural-language descriptions. Following (Wang et al. 2026a), we design the prompt to describe NPC behavior along three strategy categories: Offense, where the NPC closes the distance and attacks the player proactively; Control, where the NPC maintains distance and uses projectiles; and Defense, where the NPC reacts passively with crouching guards. Please refer to the Appendix B.2 for the detailed NPC strategy prompt. Altogether, we construct a state-aware dataset with 10,000 training clips including video frames, player actions, state information, and an NPC textual description, with a balanced distribution across different mechanics-related scenarios. 3.2 Model Architecture State and Visual Branches To jointly predict internal game states and generate mechanics-consistent visual frames, we introduce a lightweight state branch (0.76B) alongside the visual branch (5B) commonly adopted by existing game world models. This simple yet effective design shown in Fig. 3 can be readily integrated into different game world models to endow them with state modeling capability. Inspired by the action-conditioning design in world action models, we construct the state branch by perturbing the clean state sequence with Gaussian noise according to a sampled timestep : (1) where with denotes the batch size, denotes the number of latent frames, and denotes the number of state variables, which may vary across different games. During inference, the state of the first frame is provided as a condition, while the states of the remaining frames are initialized from noise and predicted by the model. The perturbed state sequence is then processed by the state encoder and state self-attention module to obtain the state representation , which captures temporal state dynamics and mechanics-related information with state token length containing dimension each. Similarly, the video branch perturbs the clean latent representation produced by the VAE encoder using independently sampled Gaussian noise : (2) where denotes the visual latent representation, consisting of visual tokens, each with feature dimension , matching the dimension of the state tokens. The perturbed visual latent is then processed by the visual self-attention module, followed by cross-attention with the text embedding, as illustrated in Fig. 3, producing the visual representation that captures both visual dynamics and semantic information from the text prompt. The state representation and visual representation are processed separately by their respective expert branches, as they correspond to intrinsically different modalities. The effectiveness of this modality-specific design is further analyzed in Sec. 4.4. In addition, the action sequence , where denotes the action dimension, is projected by an action encoder into an action embedding . The action embedding is then injected into both the visual and state branches immediately after token embedding, enabling player actions to jointly modulate visual generation and state prediction throughout the network. Mixture of Transformers After obtaining the state representation and visual representation from their respective expert branches, a joint attention module is applied for cross-modal information exchange: (3) Specifically, the visual tokens serve as queries and attend to the keys and values derived from the state tokens, allowing visual generation to incorporate mechanics-related state information. Conversely, the state tokens serve as queries and attend to the keys and values derived from the visual tokens, enabling state prediction to leverage the corresponding visual dynamics. This bidirectional interaction enables effective information exchange while preserving modality-specific representations in each branch. Loss Design StatePlay is jointly optimized for visual generation and state prediction. For the visual branch, we follow the standard flow matching objective adopted by Wan-based game world models to learn the velocity field for denoising the perturbed visual latent: (4) where denotes the velocity field predicted by the visual branch, is the perturbed visual latent at timestep , and is the corresponding ground-truth velocity under the linear flow path. For the state branch, rather than applying the same flow-matching objective used in world action models, we directly supervise the predicted states with a Smooth L1 regression loss. Compared with naively adapting flow matching to state prediction, regression is better suited to the low-dimensional, mechanics-constrained nature of game states: (5) where denotes the state decoder, and the loss is computed between the predicted state sequence and the corresponding clean state sequence . Finally, the two objectives are jointly optimized using a weighted summation: (6) where and denote the loss weights for the state and visual branches, respectively. A detailed ablation study of this design choice is provided in Sec. 4.4. 4 Experiments 4.1 Evaluation Metrics Our benchmark evaluates each generated clip from four complementary dimensions: visual quality, action control, state alignment, and mechanics fidelity. All metrics are evaluated on an additional test set of 100 generated samples, evenly covering various mechanics categories. • Visual Quality: Evaluates frame-level and perceptual fidelity by comparing generated videos with reference game-engine outputs: SSIM (Wang et al. 2004): Measures structural similarity after aligning frame size and crop, capturing scene layout, character shapes, and local image structure. LPIPS (Zhang et al. 2018): Measures perceptual similarity using deep visual features. • Action Control: Evaluates how well the generated gameplay follows the commanded player actions. It measures controllability at both the movement and combat levels: Movement Accuracy (Move-Acc): Quantifies alignment between the generated character movement and the input movement direction. We track the character using SAM2.1 (Ravi et al. 2025) and Grounding DINO (Liu et al. 2024), and count the movement as successful when the observed displacement satisfies predefined thresholds in a normalized coordinate space. Attack Accuracy (Att-Acc): Measures the consistency between the generated attack behavior and the intended attack command. We evaluate this using ClipAttackNet, a custom 6-way frame-level attack classifier based on ResNet-18 (He et al. 2016) and a 4-layer dilated TCN (Bai, Kolter, and Koltun 2018), trained on about 5k clips with a confidence threshold of 0.7. • State Alignment: Measures the accuracy of the generated game states, including the timer, player HP, opponent HP, and both skill meters. For each state variable, we compute the normalized prediction error against the target state trace and report the average distance. We further define the state alignment score as . • Mechanics Fidelity: Assesses the correctness of mechanics in generated clips. We use Gemini-3.1-Pro (Google DeepMind 2026) and GPT-5.5 (OpenAI 2026) as visual judges with a strict prompt and ground-truth reference images, comparing identified mechanics against the target label. This includes, but is not limited to, verifying the correct winner and valid super art execution. Gemini-3.1-Pro processes each video directly, whereas GPT-5.5 evaluates 24 uniformly sampled frames due to limited video input support. Results are reported as mean standard deviation over three runs for more reliable evaluation. Please refer to Appendix D for details. 4.2 Implementation Details We adopt Wan2.2-TI2V-5B (Wan et al. 2025) as the base video diffusion model, together with the Wan2.2 VAE and the UMT5-XXL text encoder (Chung et al. 2023). The model is trained on our collected dataset of 5-second video clips, where each clip is annotated with player actions, game states, and textual descriptions. Our model is trained for steps using a batch size of and a learning rate of . All video frames are resized to . The hyperparameters , , , , , , , and are set to , , , , , , , and , respectively. 媒体内容 · 前往原文查看 Method Visual Quality Action Control (%) State Alignment Mechanics Fidelity (%) SSIM LPIPS Move-Acc Att-Acc Gemini-3.1-Pro GPT-5.5 Zero-shot Evaluation Matrix-Game 3.0 (Wang et al. 2026b) – – – LingBot-World (Team et al. 2026) – – – LingBot-World 2.0 (Gao et al. 2026) – – – HY-World 1.5 (Sun et al. 2025) – – – ReactiveGWM (Wang et al. 2026a) – – – State-aware Fine-tuning Matrix-Game 3.0 (Wang et al. 2026b) – HY-World 1.5 (Sun et al. 2025) – ReactiveGWM (Wang et al. 2026a) – \rowcolorcyan!6 StatePlay Table 1: Quantitative comparison of game world models under zero-shot evaluation and state-aware fine-tuning. Bold values indicate the best performance, while “–” denotes metrics that are not applicable to a given method. 媒体内容 · 前往原文查看 Coupling Style State Loss Visual Quality Action Control (%) State Alignment Mechanics Fidelity (%) Shared MoT FM Regression SSIM LPIPS Move-Acc Att-Acc Gemini-3.1-Pro GPT-5.5 ✓ ✓ ✓ ✓ ✓ ✓ \rowcolorcyan!6 ✓ ✓ Table 2: Comparison of different coupling styles and state loss designs for state-aware world modeling. 4.3 Comparisons with Game World Models Zero-shot Evaluation StatePlay is compared with several state-of-the-art game world models in the zero-shot setting first. To ensure a fair comparison, all baseline models are evaluated using their original released architectures and pretrained weights without modification. Since existing game world models neither support our state representation nor share the same action space, we report only visual quality and mechanics fidelity. As shown in Tab. 1, Matrix-Game 3.0 (Wang et al. 2026b), LingBot-World (Team et al. 2026), LingBot-World 2.0 (Gao et al. 2026) and HY-World 1.5 (Sun et al. 2025) produce relatively poor visual quality. Although ReactiveGWM (Wang et al. 2026a), trained on an in-domain dataset, achieves substantially better visual quality than the other zero-shot baselines, its mechanics fidelity remains below 50% for both Gemini and GPT evaluation. This result indicates that current game world models can generate visually plausible gameplay but still struggle to faithfully preserve the game rules. State-aware Fine-tuning We further fine-tune the baseline models on our state-aware dataset to assess whether mechanics-consistent gameplay can emerge from visual supervision alone. To the best of our knowledge, existing open-source game world models neither explicitly model nor predict internal game states; therefore, the state alignment metric is not applicable and is omitted. We also omit fine-tuning results for the LingBot-World model family because they contain over 24B parameters and require substantially more training memory than our 5.75B StatePlay model. As shown in Tab. 1, StatePlay achieves a state alignment score of 0.947, demonstrating accurate prediction of internal game states. More importantly, it attains the highest mechanics fidelity, achieving 82.3% under Gemini evaluation and 78.3% under GPT evaluation, consistently outperforming all baselines over three runs across both evaluators. These results indicate that jointly learning state prediction and frame generation effectively guides the model toward generating gameplay that better follows the underlying game mechanics. Furthermore, introducing explicit state modeling does not compromise the core capabilities of game world models. While significantly improving mechanics fidelity, StatePlay maintains comparable or better visual quality and action control than existing baselines. Overall, comparisons in both zero-shot and fine-tuned settings show that existing game world models can generate visually plausible and controllable gameplay but struggle to preserve state-dependent rules when relying on visual modeling alone. StatePlay addresses this limitation by explicitly modeling internal game states and coupling state prediction with frame generation. 4.4 Ablation Study Figure 4: Qualitative comparison of three example rollouts (a)–(c) generated by the best-performing baseline, ReactiveGWM, and StatePlay. Red bounding boxes mark rule-violating rollouts, whereas green bounding boxes mark mechanics-consistent rollouts produced by our method. Please zoom in for better visibility. Coupling Style To evaluate the effectiveness of the MoT-style backbone for state-aware world modeling, we compare it with another common architectural choice in world action models: the shared-style backbone (Hou et al. 2026). In the shared-style design, frame tokens and state tokens are processed by a unified backbone, so visual prediction and state prediction are jointly modeled in a common latent space. In contrast, the MoT-style design keeps video and state modeling partially specialized through separate expert branches, while still enabling cross-modal interaction through shared joint attention. From Tab. 2, the performance gap is most pronounced in mechanics fidelity, where the MoT-style backbone outperforms the shared-style backbone by 17.6%. This improvement suggests that modality-specific structure is important for state-aware world modeling. Although visual dynamics and game states are tightly coupled, they correspond to intrinsically different representations: video prediction focuses on high-dimensional visual details, while state prediction follows lower-dimensional but rule-sensitive temporal dynamics. Forcing both modalities into a fully shared latent space may interfere with optimization and weaken mechanics modeling. By contrast, the MoT-style backbone allows each modality to preserve specialized representations while maintaining sufficient interaction between visual generation and state prediction. State Loss Flow matching is commonly adopted in world action models to jointly train frame generation and action prediction. However, we observe that game states have different properties from actions. Actions are usually continuous control signals and may vary freely according to the policy, whereas game states are compact, rule-governed variables with structured temporal patterns. For example, health points typically decrease after attacks, while skill meters usually increase gradually and reset after special moves. These differences make flow matching less suitable for state prediction. Although effective for high-dimensional visual generation, flow matching introduces random perturbations to low-dimensional state variables and requires the model to learn a transport vector field over their distributions. This formulation may unnecessarily complicate state prediction, whose transitions are largely determined by explicit mechanics and can be modeled more directly through temporal regression. Our experimental results from Tab. 2 further show that supervising states with a direct regression loss consistently outperforms flow matching, yielding improvements of 12.1% in state alignment and 21.6% in mechanics fidelity. Additional ablation studies are provided in Appendix E. 4.5 Qualitative Visualization We compare the qualitative results of the best-performing baseline, ReactiveGWM, and our StatePlay. For a fair comparison, both methods are given the same action sequence, text prompt, initial frame, and initial state as input. As shown in Fig. 4(a), StatePlay successfully performs the super art when the skill meter is full and correctly resets the meter to 0 after execution. In contrast, ReactiveGWM fails to trigger the super art even when the skill meter reaches the required threshold. It also produces blurred visual artifacts, where the two characters become visually entangled. Fig. 4(b) shows a game-ending case where the NPC’s health point reaches zero. ReactiveGWM produces an invalid rollout in which the NPC remains standing and attacking despite having zero health, whereas StatePlay correctly generates the “You Win” result. Fig. 4(c) shows the opposite case, where the player’s health point reaches zero. ReactiveGWM incorrectly depicts the player as winning, with the player celebrating and the NPC lying on the ground, while StatePlay generates the correct losing outcome consistent with the game rule. These visualization results further show that game world models without explicit state modeling, such as ReactiveGWM, tend to overfit to pixel-level visual dynamics and fail to capture valid game mechanics. In contrast, our state-aware design enables StatePlay to predict internal states more accurately and generate mechanics-consistent rollouts. Please refer to the Appendix F for more visualization results. 5 Conclusions and Discussions In this paper, we introduce StatePlay, a state-aware game world model that incorporates explicit internal state prediction into game frame generation. Unlike previous game world models that mainly focus on visual dynamics conditioned on player actions, StatePlay models states such as timers, health points, and skill meters as part of the generative process. By coupling state prediction with frame generation, our method produces rollouts that are both visually plausible and more consistent with the underlying game mechanics. Experiments on an intuitive fighting game demonstrate that StatePlay accurately predicts internal states and improves mechanics fidelity while preserving action controllability and visual quality. These results establish the importance of state-aware modeling and provide an initial viable framework for more complete and playable game world models. However, as publicly available datasets with synchronized state, frame and action remain limited, we focus our initial evaluation on Street Fighter 3. Broader validation across game genres requires larger and more diverse state-aware datasets. Given the benefits of state modeling for mechanics-consistent generation, we hope this work encourages the development of datasets with explicit state annotations across a wider range of games, enabling state-aware game world models to scale and generalize across diverse rule systems. References Bai, Kolter, and Koltun (2018) Bai, S.; Kolter, J. Z.; and Koltun, V. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271. Black et al. (2025) Black, K.; Brown, N.; Darpinian, J.; Dhabalia, K.; Driess, D.; Esmail, A.; Equi, M. R.; Finn, C.; Fusai, N.; Galliker, M. Y.; et al. 2025. : a Vision-Language-Action Model with Open-World Generalization. In 9th Annual Conference on Robot Learning. Bruce et al. (2024) Bruce, J.; Dennis, M. D.; Edwards, A.; Parker-Holder, J.; Shi, Y.; Hughes, E.; Lai, M.; Mavalankar, A.; Steigerwald, R.; Apps, C.; et al. 2024. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning. Capcom (1998) Capcom. 1998. Street Fighter Alpha 3. Arcade. Chung et al. (2023) Chung, H. W.; Constant, N.; Garcia, X.; Roberts, A.; Tay, Y.; Narang, S.; and Firat, O. 2023. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151. Gao et al. (2026) Gao, Z.; Wang, Q.; Zhu, J.; Chen, J.; Liu, Z.; Bai, Q.; Wang, J.; Yuan, Y.; Wang, H.; Lu, Y.; Cheng, K. L.; Zhang, H.; Gao, J.; Feng, T.; Liu, Y.; Yao, Y.; Xu, Y.; Zhu, X.; Shen, Y.; and Ouyang, H. 2026. Infinite Worlds with Versatile Interactions. arXiv:2607.07534. Google DeepMind (2026) Google DeepMind. 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/models/model-cards/gemini-3-1-pro/. Accessed: 2026-07-28. He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778. Hou et al. (2026) Hou, B.; Li, G.; Jia, J.; An, T.; Guo, X.; Leng, S.; Geng, H.; Ze, Y.; Harada, T.; Torr, P.; et al. 2026. World model for robot learning: A comprehensive survey. arXiv preprint arXiv:2605.00080. Hu et al. (2022) Hu, A.; Corrado, G.; Griffiths, N.; Murez, Z.; Gurau, C.; Yeo, H.; Kendall, A.; Cipolla, R.; and Shotton, J. 2022. Model-based imitation learning for urban driving. Advances in Neural Information Processing Systems, 35: 20703–20716. Hu et al. (2026) Hu, Y.; Zhang, J.; Luo, Y.; Guo, Y.; Chen, X.; Sun, X.; Feng, K.; Lu, Q.; Chen, S.; Zhang, Y.; et al. 2026. Bagelvla: Enhancing long-horizon manipulation via interleaved vision-language-action generation. arXiv preprint arXiv:2602.09849. Kim et al. (2026) Kim, M. J.; Gao, Y.; Lin, T.-Y.; Lin, Y.-C.; Ge, Y.; Lam, G.; Liang, P.; Song, S.; Liu, M.-Y.; Finn, C.; et al. 2026. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163. Kim et al. (2024) Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E. P.; Sanketi, P. R.; Vuong, Q.; et al. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. In 8th Annual Conference on Robot Learning. Li et al. (2026a) Li, Z.; Meng, Z.; Shi, S.; Peng, W.; Wu, Y.; Zheng, B.; Li, C.; and Zhang, K. 2026a. Wildworld: A large-scale dataset for dynamic world modeling with actions and explicit state toward generative arpg. arXiv preprint arXiv:2603.23497. Li et al. (2026b) Li, Z.; Meng, Z.; Shi, S.; Zhai, M.; Tan, J.; Li, C.; and Zhang, K. 2026b. From Pixels to States: Rethinking Interactive World Models as Game Engines. arXiv:2607.14076. Liu et al. (2024) Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; et al. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, 38–55. Springer. Mao et al. (2026) Mao, X.; Li, Z.; Li, C.; Xu, X.; Ying, K.; and Zhang, K. 2026. Yume1. 5: A text-controlled interactive world generation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7752–7761. OpenAI (2026) OpenAI. 2026. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/. Accessed: 2026-07-28. Poliquin (2026) Poliquin, M. 2026. Stable Retro: A Maintained Fork of OpenAI’s Gym Retro. Accessed: 2026-07-07. Ravi et al. (2025) Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; et al. 2025. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations, volume 2025, 28085–28128. Seedance et al. (2026) Seedance, T.; Chen, D.; Chen, L.; Chen, X.; Chen, Y.; Chen, Z.; Chen, Z.; Cheng, F.; Cheng, T.; Cheng, Y.; et al. 2026. Seedance 2.0: Advancing video generation for world complexity. arXiv preprint arXiv:2604.14148. Shen et al. (2026) Shen, T.; Bahmani, S.; He, K.; Srinivasan, S. G.; Cao, T.; Ren, J.; Li, R.; Wang, Z.; Sharp, N.; Gojcic, Z.; et al. 2026. Lyra 2.0: Explorable generative 3d worlds. arXiv preprint arXiv:2604.13036. Sun et al. (2025) Sun, W.; Zhang, H.; Wang, H.; Wu, J.; Wang, Z.; Wang, Z.; Wang, Y.; Zhang, J.; Wang, T.; and Guo, C. 2025. Worldplay: Towards long-term geometric consistency for real-time interactive world modeling. arXiv preprint arXiv:2512.14614. Team et al. (2025) Team, K.; Chen, J.; Ci, Y.; Du, X.; Feng, Z.; Gai, K.; Guo, S.; Han, F.; He, J.; He, K.; et al. 2025. Kling-Omni Technical Report. arXiv preprint arXiv:2512.16776. Team et al. (2026) Team, R.; Gao, Z.; Wang, Q.; Zeng, Y.; Zhu, J.; Cheng, K. L.; Li, Y.; Wang, H.; Xu, Y.; Ma, S.; et al. 2026. Advancing Open-source World Models. arXiv preprint arXiv:2601.20540. Tong et al. (2026) Tong, Z.; Lai, H.; Wang, Z.; Xing, Z.; Cheng, K.; Xu, H.; Pu, Z.; Zhu, S.; Feng, R.; Zhao, J.; et al. 2026. SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models. arXiv preprint arXiv:2605.23345. Wan et al. (2025) Wan, T.; Wang, A.; Ai, B.; Wen, B.; Mao, C.; Xie, C.-W.; Chen, D.; Yu, F.; Zhao, H.; Yang, J.; et al. 2025. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Wang et al. (2024) Wang, X.; Zhu, Z.; Huang, G.; Chen, X.; Zhu, J.; and Lu, J. 2024. Drivedreamer: Towards real-world-drive world models for autonomous driving. In European conference on computer vision, 55–72. Springer. Wang et al. (2004) Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600–612. Wang et al. (2026a) Wang, Z.; Chen, D.; Xing, Z.; Tong, Z.; Zhang, Y.; Yang, X.; and Jin, Y. 2026a. ReactiveGWM: Steering NPC in Reactive Game World Models. arXiv preprint arXiv:2605.15256. Wang et al. (2026b) Wang, Z.; Liu, Z.; Li, J.; Huang, K.; Xu, B.; Kang, F.; An, M.; Wang, P.; Jiang, B.; Wei, Y.; et al. 2026b. Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory. arXiv preprint arXiv:2604.08995. Wu et al. (2026) Wu, H.; Yu, J.; Zou, Y.; and Liu, X. 2026. Multiworld: Scalable multi-agent multi-view video world models. arXiv preprint arXiv:2604.18564. Ye et al. (2026) Ye, S.; Ge, Y.; Zheng, K.; Gao, S.; Yu, S.; Kurian, G.; Indupuru, S.; Tan, Y. L.; Zhu, C.; Xiang, J.; et al. 2026. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. Yu et al. (2025) Yu, J.; Qin, Y.; Wang, X.; Wan, P.; Zhang, D.; and Liu, X. 2025. Gamefactory: Creating new games with generative interactive videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11590–11599. Zhang et al. (2018) Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586–595. Zheng et al. (2024) Zheng, W.; Chen, W.; Huang, Y.; Zhang, B.; Duan, Y.; and Lu, J. 2024. Occworld: Learning a 3d occupancy world model for autonomous driving. In European conference on computer vision, 55–72. Springer. Zhu et al. (2026) Zhu, S.; Peng, Q.; Pu, Z.; Shu, Z.; Ke, X.; Xing, Z.; Tong, Z.; Wang, Z.; Cui, X.; Wang, H.; et al. 2026. Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models. arXiv preprint arXiv:2605.18601. A Experimental Model Information Tab. 3 summarizes the model configuration used in our experiments. StatePlay adds a lightweight 0.76B-parameter state branch while consistently outperforming the game world model baselines. Since most models are built on base diffusion models, text encoders, and VAEs from the same model family, these results further demonstrate the effectiveness of our MoT-style architecture for state-aware game modeling. 媒体内容 · 前往原文查看 Model Params. Base Model Text Encoder VAE Matrix-Game 3.0 6.312B Wan2.2-TI2V-5B UMT5-XXL Wan2.2 LingBot-World 37.089B Wan2.1-I2V-A14B with high-/low-noise DiTs UMT5-XXL Wan2.1 LingBot-World 2.0 24.352B Wan2.1-I2V-A14B UMT5-XXL Wan2.1 HY-World 1.5 18.11B HunyuanVideo-1.5 480P-I2V Qwen2.5-VL-7B-Instruct HYVideo-1.5 ReactiveGWM 5.001B Wan2.2-TI2V-5B UMT5-XXL Wan2.2 StatePlay 5.759B Wan2.2-TI2V-5B UMT5-XXL Wan2.2 Table 3: Architecture configurations of the evaluated game world models. B State-aware Dataset Construction B.1 Mechanics Distribution Balancing Criteria To ensure balanced coverage of diverse mechanics scenarios, we construct the training dataset according to the distribution shown in Tab. 4. A clip is assigned to a specific category only when it satisfies both the corresponding visual and state conditions, ensuring consistency between the observed gameplay and the underlying mechanics. We collect 1,000 clips for each of the four state-critical categories and 6,000 clips for normal gameplay, resulting in 10,000 training samples. 媒体内容 · 前往原文查看 Category # Visual condition State condition Result Win 1k “You Win” in final frame Player HP , NPC HP Result Lose 1k “You Lose” in final frame Player HP , NPC HP Macro Success 1k Super art is executed Macro input; Player skill meter Macro Fail 1k No super art is executed Macro input; Player skill meter Normal 6k No terminal or super art event None Total 10k 40% state-critical, 60% normal Table 4: Dataset categories and classification criteria. B.2 NPC Strategy Annotation To encourage NPCs to engage players strategically in generated rollouts, we follow ReactiveGWM (Wang et al. 2026a) and use Gemini-3.1-Pro (Google DeepMind 2026) to classify NPC behavior into three strategies: offense, control, and defense. The detailed annotation prompt is shown in Fig. 5. The resulting strategy label is included in the training dataset alongside the corresponding video clip, player actions, and game states. 媒体内容 · 前往原文查看 Figure 5: NPC strategy annotation prompt for classifying fighting behaviors and incorporating diverse strategy descriptions into the dataset. Figure 6: Evaluation prompt for mechanics fidelity (top), with three ground-truth reference images (bottom). Figure 7: Qualitative comparison of three example rollouts (a)–(c) generated by the best-performing baseline, ReactiveGWM, and StatePlay. Please zoom in for better visibility. C Failure Cases Although StatePlay accurately predicts the internal states required for valid mechanics, pixel-level indicators such as health bars and skill meters may still show occasional inconsistencies. Visual quality can also degrade when multiple mechanics occur simultaneously, such as a super art coinciding with the end of a match. Stronger visual generation capabilities may help address these limitations. D Details of Mechanics Fidelity Evaluation To robustly evaluate mechanics fidelity, we prompt Gemini-3.1-Pro and GPT-5.5 with explicit visual rules, a fixed decision priority, and reference images to determine whether each generated rollout follows the ground-truth mechanics. Win and loss are recognized only when the corresponding result screen is visible, while macro success requires a clearly executed super art that successfully hits the opponent. The complete prompt and reference images are shown in Fig. 6. E State Input Comparisons Since StatePlay uses a regression objective for state prediction, we compare two ways of constructing the state input during training: broadcasting the initial clean state across all timesteps and adding noise to the state sequence. As shown in Tab. 5, the noise-based input consistently outperforms the broadcasting baseline in both state alignment and mechanics fidelity. We hypothesize that broadcasting the same clean state across all timesteps produces an overly static initialization, which hinders the learning of temporally evolving frame dynamics. Therefore, following prior world action models (Hu et al. 2026; Ye et al. 2026), we perturb both the video latents and raw states during training. 媒体内容 · 前往原文查看 State Input State Alignment Mechanics Fidelity (%) Broadcast Noise Gemini GPT ✓ \rowcolorcyan!6 ✓ Table 5: Comparison of different state input designs. F More Qualitative Comparisons We provide additional qualitative comparisons in Fig. 7. ReactiveGWM and StatePlay are fine-tuned on the same state-aware dataset for steps. Despite this identical training setup, ReactiveGWM, which models only visual observations without explicitly predicting game states, frequently generates rollouts that violate the underlying game mechanics. Specifically, Fig. 7(a) shows that it fails to execute a super art even when the skill meter is full. In Fig. 7(b), it does not display the game result after the NPC’s health reaches zero. In Fig. 7(c), although the “You Lose” message appears, the subsequent frames become blurred and distorted instead of transitioning to a proper game-ending screen. In contrast, StatePlay generates mechanics-consistent rollouts by explicitly modeling the underlying game states.