MoVerse: 基于全景高斯支架的实时视频世界模型
阅读原文· arxiv.orgMoVerse 能从单张窄视野图像生成可交互漫游的实时视频世界模型。它将世界构建与观察渲染分离:先用拓扑感知扩散将输入扩展为重力对齐的 360° 全景图,再通过全景几何感知残差预测将其提升为持久 3D Gaussian scaffold,最后由 Gaussian 条件视频渲染器沿用户指定相机轨迹输出逼真视频。渲染器采用双向扩散教师蒸馏为因果自回归学生,实现有限延迟流式输出。在单个 NVIDIA RTX 4090 GPU 上支持 8 FPS 的实时场景漫游。
We present MoVerse, a real-time video world model that creates an interactively navigable scene from a single narrow-field-of-view image. This setting is challenging because the input observes only a small fraction of the environment, while interactive roaming requires a complete surrounding world, persistent geometry, controllable camera motion, and temporally coherent high-fidelity observations. MoVerse addresses this problem by separating world construction from observation rendering. It first expands the input into a gravity-aligned 360^circ panorama with topology-aware diffusion, closing the missing field of view before 3D reasoning. It then lifts the panorama into a persistent 3D Gaussian scaffold using panoramic geometry-aware residual prediction, yielding a dense and directly renderable spatial memory. Finally, a Gaussian-conditioned video renderer translates scaffold renderings along user-specified camera trajectories into photorealistic video. To make this renderer practical for interaction, we train a bidirectional diffusion teacher for high-quality conditional rendering and distill it into a causal autoregressive student for bounded-latency streaming. This design combines the controllability and long-range consistency of explicit 3D representations with the perceptual quality of generative video models. MoVerse supports real-time scene roaming at 8~FPS on a single NVIDIA RTX~4090 GPU, demonstrating a practical path toward single-image world creation with interactive video output.