# 面向视频到语音生成的分层编解码器扩散模型

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-04-17 08:00
- AIHOT 链接：https://aihot.virxact.com/items/cmo75l42m01cvsli5i9rxki62
- 原文链接：https://arxiv.org/abs/2604.15923

## AI 摘要

研究团队提出分层编解码器扩散Transformer模型 HiCoDiT，利用 RVQ 编解码器的分层结构解决现有视频到语音生成方法忽视语音层级特性的问题。该方法通过低级块基于唇形同步运动与面部身份建模说话人感知语义，高级块利用面部表情调节细粒度韵律动态，并引入双尺度自适应实例层归一化实现从粗到细的条件控制。实验表明，该模型在语音保真度和表现力上显著优于基线方法，代码与演示已开源。

## 正文

Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.