Go-with-the-Track: 视频合成与运动控制与点追踪
阅读原文· arxiv.orgGo-with-the-Track将精确合成与运动控制统一在单一视频扩散Transformer中,通过联合多个参考图像和参考锚定点追踪实现。模型引入空间感知点追踪嵌入,利用坐标MLP和时序池化编码完整点轨迹序列,再通过轻量适配器注入模型,避免像素与补丁分辨率不匹配及下采样导致的信息损失。混合训练策略在动态、静态及合成视频数据集上联合训练以增强运动可控性。实验表明,该模型能支持多参考条件视频生成、点追踪驱动合成,并对静态与动态场景提供相机控制。
Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/