像素之前的表示:语义引导的分层视频预测
阅读原文· arxiv.org研究团队推出Re2Pix分层视频预测框架,将预测任务分解为语义表示预测与表示引导的视觉合成两阶段。该方法先在冻结视觉基础模型的特征空间中预测未来场景结构,再基于这些表示通过潜在扩散模型渲染真实帧。针对训练与推理时的表示不匹配问题,引入嵌套dropout和混合监督两种条件策略。在自动驾驶等复杂动态环境基准测试中,该语义优先设计显著提升了时间语义一致性、感知质量和训练效率。
Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix