MVTrack4Gen:多视角点跟踪作为4D视频生成的几何监督
阅读原文· arxiv.orgMVTrack4Gen提出运动感知训练框架,将多视角点跟踪作为额外几何与运动监督信号,用于仅依赖相机条件的新视角视频扩散模型。关键发现是特定注意力层编码了跨视角和时序上的几何对应关系,对齐偏差会导致运动不一致。通过将这些特征路由到辅助多视角跟踪头并联合训练点跟踪目标,MVTrack4Gen增强运动感知对应,使模型更好地保持参考视角的运动和跨视角几何一致性。在多个基准上,该方法达到最优几何一致性和有竞争力的相机精度。
Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.