SwanVoice:面向独白与对话的富有表现力的零样本文本转语音模型
阅读原文· arxiv.orgSwanVoice 是一个支持 1-4 位说话人的零样本文本转语音(TTS)模型,旨在解决现有方法在合成富有表现力的长对话时推理成本高、声学一致性和情感连贯性差的问题。模型基于 SwanData-Speech 数据集构建,采用 25Hz VAE 与带停顿感知符号的原始文本条件处理,并结合具有说话人轮次条件的 flow-matching DiT。训练从独白数据开始,逐步过渡到真实对话数据,并使用 DiffusionNFT 进行后训练。在 SwanBench-Speech 评测中,SwanVoice 在独白和对话设置下的丰富性与层次性分数均优于所有开源基线,但内容准确性仍是主要限制。音频 demo 已上线。
Zero-shot text-to-speech (TTS) has improved substantially for single-speaker synthesis, yet expressive long-form multi-speaker dialogue remains difficult. A common workaround is to synthesize each turn with a monologue TTS model and stitch the outputs together. This adds inference cost and often breaks acoustic consistency, conversational coherence, and affective continuity across turns. Recent dialogue TTS systems have begun to address this setting, but they still struggle to keep expressive coherence, controllable speaker switching, and monologue quality at the same time. We present SwanData-Speech and SwanVoice. SwanData-Speech builds monologue and dialogue corpora from in-the-wild audio, using Swan Forced Aligner for pause-aware word-level alignment and RobustMegaTTS3 for pronunciation-hard cases. Built on these data, SwanVoice is a zero-shot TTS model for 1--4 speakers, combining a 25 Hz VAE, raw-text conditioning with pause-aware symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning. Training starts from monologue speech, moves through mixed and real dialogue data, and then uses DiffusionNFT post-training with phone-level and speaker-similarity rewards. On SwanBench-Speech, SwanVoice obtains higher richness and hierarchy scores than all evaluated open-source baselines in both monologue and dialogue settings, while content accuracy remains the main limitation. Audio demos are available at https://swanaigc.github.io//#swanvoice.