# Video2LoRA：面向视觉语言模型的参数化视频内化方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-03 08:00
- AIHOT 分数：68
- AIHOT 链接：https://aihot.virxact.com/items/cmq0bomkt04phsltr9nzj446z
- 原文链接：https://arxiv.org/abs/2606.04351

## AI 摘要

Video2LoRA通过感知器超网络读取冻结视觉语言模型编码视频时的逐层中间表示，单次前向传播生成LoRA适配器，无需迭代梯度更新。在SmolVLM2 500M和2.2B上训练后，同一冻结VLM仅从适配器回答查询，上下文中零视觉token。在五个字幕基准和八个视频问答基准配对中，Video2LoRA非劣效且等价于直接视频上下文推理。虽仅用12帧384px训练，但稳定支持1024帧和1024px，将回答时视觉token负载减少最高1500倍，查询TTFT减少6–80倍。非重叠视频段独立生成的适配器可在秩空间中组合。

## 正文

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.