作者：Andrei Atanov*+、Jesse Allardice、Roman Bachmann+、Oğuzhan Fatih Kar+、Devon Hjelm、David Griffiths、Peter Fu、Afshin Dehghan、Amir Zamir+
视觉 token 编码器将高维原始像素映射为压缩表示，供下游建模使用。除了压缩功能外，token 编码器还决定了哪些信息被保留以及信息如何组织。视频 token 编码的一种事实标准方法是将视频表示为时空三维 token 网格，每个 token 捕获原始信号中对应的局部信息。这就要求消费这些 token 的下游模型（例如文生视频模型）必须学会“逐像素”预测所有低层细节，而不考虑视频本身的固有复杂度，从而导致较高的学习复杂度。我们提出了 VideoFlexTok，它采用一种由粗到精的结构化方式，将视频表示为可变长度的 token 序列——其中前几个 token（涌现性地）捕获抽象信息，如语义和运动，而后面的 token 则添加精细细节。生成流解码器能够从任意数量的 token 中重建出逼真的视频。这种表示结构允许根据下游需求调整 token 数量，并在相同预算下编码比基线方法更长的视频。我们在类别条件和文生视频生成任务上评估了 VideoFlexTok，结果表明，与三维网格 token 相比，它能实现更高效的训练，例如，使用小 5 倍的模型（11 亿参数对比 52 亿参数）即可达到相当的生成质量（gFVD 和 ViCLIP 分数）。最后，我们展示了 VideoFlexTok 如何通过仅使用 672 个 token（比同类三维网格 token 编码器少 8 倍）在 10 秒 81 帧的视频上训练文生视频模型，从而在不产生过高计算成本的情况下实现长视频生成。
- + 瑞士洛桑联邦理工学院（EPFL）
- ** 在苹果公司期间完成的工作
相关阅读与更新。
TrajTok：学习轨迹 Token 以提升视频理解能力
视频模型中的分词化（通常通过分块处理）会产生大量冗余的模型 token。这严重限制了视频处理的效率和可扩展性。尽管近期基于轨迹的分词器通过将视频时长与 token 数量解耦，提供了一种有前景的解决方案，但它们依赖于复杂的外部分割和跟踪流程，这些流程速度缓慢且与任务无关。我们提出了 TrajTok，一种端到端的视频分词器模块……
FlexTok：将图像重采样为可变长度的一维 token 序列
2025 年 2 月 19 日 研究领域 计算机视觉
本研究与洛桑联邦理工学院（EPFL）合作完成。
图像分词化通过提供压缩的离散表示（其处理效率高于原始像素），推动了自回归图像生成的重大进展。虽然传统方法采用二维网格分词化，但近期像 TiTok 这样的方法已证明，通过消除网格结构，一维分词化能够实现高质量的生成……
探索机器学习领域的机遇。
AuthorsAndrei Atanov+**, Jesse Allardice, Roman Bachmann+, Oğuzhan Fatih Kar+, Devon Hjelm, David Griffiths, Peter Fu, Afshin Dehghan, Amir Zamir+
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details “pixel-by-pixel” irrespective of the video’s inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner — where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
- + Swiss Federal Institute of Technology Lausanne (EPFL)
- ** Work done while at Apple
Related readings and updates.
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that…
FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
February 19, 2025research area Computer Vision
This work was done in collaboration with Swiss Federal Institute of Technology Lausanne (EPFL).
Image tokenization has enabled major advances in autoregressive image generation by providing compressed, discrete representations that are more efficient to process than raw pixels. While traditional approaches use 2D grid tokenization, recent methods like TiTok have shown that 1D tokenization can achieve high generation quality by eliminating grid…