# AVTok：面向整体音频-视频生成的一维统一分词器

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-29 08:00
- AIHOT 分数：44
- AIHOT 链接：https://aihot.virxact.com/items/cmr1ghdum02mrslnl0rrpasnu
- 原文链接：https://arxiv.org/abs/2606.30811

## AI 摘要

AVTok 是一种新颖的统一分词器，专为整体音频-视频生成设计。它采用双流 Transformer 架构，包含共享编码器-解码器和模态特定的可学习查询，将音频-视频对高效编码为紧凑的一维潜在表示并共享同一码本。为应对异质信息不平衡，研究者设计了分层训练策略，逐步重建各模态。实验表明，AVTok 在音频-视频重建及下游任务（音频到视频、视频到音频、类别条件联合生成）中均表现优异，为构建统一音视频大语言模型提供了潜在方向。

## 正文

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.