ActWorld:通过动作感知记忆从可探索走向可交互的世界模型
阅读原文· arxiv.org现有交互式世界模型的动作词汇局限于导航(行走、转向、环顾),缺乏物体交互。ActWorld 在分块自回归框架中扩展导航生成器,支持 rollout 过程中的物体交互。它解决数据瓶颈(缺乏带密集标签的人-物交互数据)和记忆瓶颈(历史压缩丢弃因果决定物体状态的帧)。团队构建 100K 交互视频数据集,每条视频通过链式推理生成每块描述;引入分层动作感知记忆设计,按交互重要性路由历史压缩,辅以持久记忆库维护事件更新和物体身份 token。实验表明,单个模型同时支持灵活导航与丰富物体交互,在不牺牲视点控制的前提下显著提升交互逼真度。
Interactive world models aim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends prior navigation-centric generators to support mid-rollout object interaction within a chunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interaction data with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards the event-transition frames that causally determine subsequent object states, leading to an action-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated with per-chunk captions via chain-of-thought reasoning. On the model side, we introduce a hierarchical action-aware memory design that routes history compression by interaction importance, complemented by a persistent memory bank that maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.