FlowMimic:无需遮罩的在线视频编辑数据生成与模态模仿框架

HuggingFace Daily Papers(社区热门论文)·2026-07-20 08:00·46天前
AI 导读

FlowMimic提出像素对时序扭曲流场,直接从图像编辑样本实时生成对应视频编辑样本,无需人工遮罩标注或I2V模型合成。该方法通过模态模仿生成损失和编辑损失对齐图像与视频模态的输出分布,并引入指代表达分割等感知任务及编辑区域感知的隐层与注意力损失,使模型内化语言驱动的视觉编辑能力。

HuggingFace Daily Papers(社区热门论文)
43AI 编辑部评分,满分 100

FlowMimic:无需遮罩的在线视频编辑数据生成与模态模仿框架

2026-07-20 08:00· 46天前
AI 导读

FlowMimic提出像素对时序扭曲流场,直接从图像编辑样本实时生成对应视频编辑样本,无需人工遮罩标注或I2V模型合成。该方法通过模态模仿生成损失和编辑损失对齐图像与视频模态的输出分布,并引入指代表达分割等感知任务及编辑区域感知的隐层与注意力损失,使模型内化语言驱动的视觉编辑能力。

In line with the prevailing direction of vision research, we explore the integration of both generation and editing capabilities for video and image modalities within a single model. Current approaches to collecting video editing data typically depend on labour-intensive, time-consuming curated procedures--involving object mask annotation, the use of error-introducing pair synthesis via I2V model and ControlNet-like guidance, and VLM-based quality filtering or refinement--and demonstrate limited task scalability. As a result, the diversity of editing tasks remains substantially narrower than that available for image editing models. We develop a pixel-pair temporal warped flow field that can directly generate corresponding video editing samples in real time from image editing samples, and we demonstrate across multiple levels of video editing tasks that a model can learn video editing using only such data. We regard the image modality as a particular form of the video modality. Accordingly, we design a modality mimic generation loss and a modality mimic editing loss to relatively align the capabilities--and thereby the output distributions--of the two modalities through mutual imitation. Moreover, language-based visual editing entails the comprehension of the editing instruction and the reference visual content, the localization of the region corresponding to that instruction within the reference visual contents, and the modification of that region alone. Existing approaches predominantly rely on external aids, such as fine-tuning an additional MLLM or explicitly supplying a mask sequence as auxiliary input during inference. In contrast, we aspire for the model to internalize this capability. To that end, we introduce sense-related tasks--for instance, referring expression segmentation--along with corresponding editing-region-aware latent-level loss and attention-level loss.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org