Sreyan Ghosh
Arushi Goel
Kaousheik Jayakumar
Lasha Koroshinadze
Nishit Anand
Zhifeng Kong
Siddharth Gururani
Sang-gil Lee
Jaehyeon Kim
Aya Aljafari
Chao-Han Huck Yang
Sungwon Kim
Ramani Duraiswami
Dinesh Manocha
Mohammad Shoeybi
Bryan Catanzaro
Ming-Yu Liu
Wei Ping
NVIDIA，美国
马里兰大学，美国
项目负责人。排名顺序由抛硬币决定。
模型
摘要
我们推出 Audio Flamingo Next（AF-Next），这是 Audio Flamingo 系列中最新一代、能力最强的大型音频语言模型，旨在提升对语音、环境音和音乐的理解与推理能力。与 Audio Flamingo 3 相比，AF-Next 引入了：（i）更强的底层音频-语言模型，显著提升了各类音频理解任务的准确率；（ii）可扩展的策略，用于构建超越现有学术基准的大规模音频理解与推理数据；（iii）支持长达 30 分钟的长音频与复杂音频输入；（iv）时序音频思维链，这是一种新的推理范式，能将中间推理步骤显式地锚定到长音频中的时间戳上，从而实现细粒度的时序对齐并提升可解释性。为实现这些能力，我们首先对 Audio Flamingo 3 进行了系统分析，以识别其在音频理解与推理方面的关键差距。随后，我们整理并扩展了新的超大规模数据集，总计超过 100 万小时，以弥补这些局限，并扩充了现有的 AudioSkills-XL、LongAudio-XL、AF-Think 和 AF-Chat 数据集。AF-Next 采用基于课程学习的训练策略，涵盖预训练、中期训练和后训练阶段。在 20 个音频理解与推理基准（包括具有挑战性的长音频任务）上的大量实验表明，AF-Next 以大幅优势超越了同等规模的开源模型，并且与更大规模的开源权重模型和闭源模型相比，具有很强的竞争力，有时甚至更胜一筹。除了基准性能之外，AF-Next 还展现出强大的实际应用价值，并能很好地迁移到未见过的任务上，凸显了其鲁棒性和泛化能力。除了所有数据、代码和方法，我们还开源了 AF-Next 的 3 个变体，包括 AF-Next-Instruct（面向问答）、AF-Next-Think（面向高级推理）和 AF-Next-Captioner（面向详细描述）。
1. 引言
音频，涵盖语音、环境声音和音乐，是人类感知世界并与世界交互的核心方式。强大的音频理解能力支撑着对话、情境感知和音乐聆听等核心功能，并赋能自动语音识别（ASR）、音频描述生成和音乐信息检索（MIR）等应用。历史上，这些问题是通过使用小型、特定任务的模型（Peng 等人，2026；Heydari 和 Duan，2021）孤立地进行研究的。最近，大规模训练的大型音频语言模型（LALM）开始统一这些任务，展现出强大的跨领域迁移能力和广泛的覆盖范围（Goel 等人，2024a；Xu 等人，2025b）。然而，与视觉语言模型（VLM）相比，开源 LALM 的规模化进展明显较慢，这进一步限制了音频在通用多模态系统以及音频生成和世界建模（Wang 等人，2025c；Kim 和 Seo，2025；Ghosh 等人，2025c）等下游工作中的作用。
一个关键障碍在于，许多开源 LALM 的开发要么是封闭的，要么与一小部分学术基准紧密耦合。虽然基准很有价值，但它们存在偏差且覆盖不完整（Kumar 等人，2025b），而音频基准尤其仍在发展之中。因此，以基准为中心的训练可能会产生在精心策划的测试集上表现良好，但在处理长时长、嘈杂且多样化的真实世界音频时泛化能力较差的模型。近期前沿系统既展示了机遇也暴露了差距：Audio Flamingo（Kong 等人，2024）和 Qwen（Chu 等人，2023a）系列模型引入了长音频理解和多轮音频对话等能力，这些能力尚未被标准化基准全面评估，这促使我们转向更能反映真实世界应用的数据和训练方案。
主要贡献。我们提出了 Audio Flamingo Next（AF-Next），这是一个完全开源¹的通用型大型音频语言模型，在 20 多项音频理解与推理基准测试中达到了业界领先水平，同时显著提升了对复杂真实世界长音频的鲁棒性。AF-Next 是迈向规模化完全开源音频理解的第一步，它超越了学术数据集和基准测试的局限，通过利用互联网规模的音频数据以及推理后训练来实现。具体而言，我们（i）通过从互联网规模来源中精选高质量数据，将训练数据扩展到学术数据集之外，重点关注那些更能反映真实部署条件的、时长较长、内容多样且声学上具有挑战性的音频；（ii）在 Audio Flamingo 任务套件中增强并拓宽模型能力，包括改进 ASR 和音频描述生成，并引入新能力，如多人对话 ASR、带时间戳的预测、长音频描述生成以及指令遵循；（iii）引入时间音频思维链，这是一种推理范式，能将中间推理步骤明确地锚定到长音频中的时间戳上。为支撑这些进展，AF-Next 采用四阶段课程进行训练，包括多轮监督微调和基于 GRPO 的强化学习，并精心设计了数据混合。总之，我们的主要贡献是： ¹ 所谓完全开源，是指模型的权重、训练数据和代码将公开发布，训练方法完全透明（与仅开源权重和闭源模型不同）。由于本工作中所用训练数据的许可和范围限制，所有发布内容将仅限研究用途许可。
-
我们推出 AF-Next，这是一款开放前沿的通才型 LALM，在多个维度上推动了音频理解与推理能力的发展。据我们所知，AF-Next 是首个将音频理解能力扩展至互联网规模数据的完全开放 LALM。在 20 多个基准测试上的广泛实验表明，它以大幅优势优于同等规模的开源模型，同时与规模大得多的开源权重模型及闭源模型相比仍极具竞争力，有时甚至超越后者，尤其是在处理长时长、复杂的真实世界音频方面。
-
我们为下一代 LALM 开发了一套可扩展的训练方案，涵盖互联网规模的数据整理、针对性能力扩展，以及面向长音频的时序基础推理。我们开源了训练和推理代码及相关技术，以支持开放 LALM 领域的未来研究。
-
我们开源了三个模型检查点：AF-Next-Instruct、AF-Next-Think 和 AF-Next-Captioner，分别面向通用问答、高级推理和详细描述生成。
2. 方法论
2.1 Audio Flamingo Next 架构
在本节中，我们将介绍为 Audio Flamingo Next 提出的架构，如图 3 所示。与 Audio Flamingo 3 和 Music Flamingo 类似，AF-Next 包含四个主要组件：i) 采用滑动窗口特征提取的音频编码器，ii) 用于将音频嵌入向量投影到大语言模型语言空间的音频投影器，iii) 纯文本预训练的大语言模型主干网络，以及 iv) 流式 TTS。以下我们提供每个组件的详细说明。
AF-Whisper 音频编码器。沿用 AF3 和 Music Flamingo 的做法，我们采用了相同的基于 Whisper 的 AF-Whisper 音频编码器，并在更大、更多样化的语料库上进行了进一步预训练，该语料库包含多语种语音和多说话人 ASR 数据。关于 AF-Whisper 的训练细节，请读者参阅 Goel 等人（2025）的论文。
特征提取。给定一段音频输入，我们首先将其重采样为 16 kHz 单声道，并使用 25 毫秒窗口和 10 毫秒步长将波形转换为 128 通道的对数梅尔频谱图。然后，该频谱图通过 AF-Whisper 以获得隐藏表示，记为 ，其中 。音频以不重叠的 30 秒片段进行处理。因此，时间分辨率 取决于音频时长和训练期间使用的最大滑动窗口数量。AF-Whisper 以 50 Hz 的频率输出特征，之后我们按照 Chu 等人（2024）的方法应用一个步长为 2 的池化层。隐藏维度为 1280。
音频适配器。为了桥接音频表示与大语言模型的文本嵌入空间，我们引入了音频适配器层，记为 。具体来说，AF-Whisper 的表示被映射为适配后的嵌入 ，这些嵌入随后与文本指令一起作为音频提示词提供给大语言模型。我们使用一个 2 层 MLP 作为音频适配器。
大语言模型。我们使用 Qwen-2.5-7B Team (2024) 作为骨干大语言模型，这是一个仅解码器的因果模型，拥有 7B 参数、36 个 Transformer 层和 16 个注意力头。我们通过额外的长上下文训练（详见第 2.2.2 节）将其上下文长度从 32k 扩展至 128k 个模型 token。与 Music Flamingo 类似，我们将原始的 RoPE 替换为旋转时间嵌入（RoTE）Goel 等人（2024b），其中旋转角度使用每个 token 的绝对时间戳而非其离散索引 来定义。具体来说，RoTE 不使用标准 RoPE 中的 ，而是使用 ，从而产生具有时间基础的位次表示。对于以固定 40 毫秒步长生成的音频 token（Radford 等人，2022；Goel 等人，2025），我们插值离散时间位置并将其输入 RoTE 模块。RoTE 是 AF-Next 的核心组件，对于时间音频思维链尤为重要，能够增强时间理解能力，尤其是在处理长音频时。我们计划在未来工作中发布使用更小和更大骨干大语言模型的其他 AF-Next 变体。
流式 TTS。为了支持语音到语音的交互，与 AF3 类似，AF-Next 集成了一个流式 TTS 模块。该模块实现为一个仅解码器的 Transformer 架构，根据来自大语言模型的输入子词文本 token 和先前生成的音频 token 来预测下一个音频 token。更多细节，我们请读者参阅 Goel 等人（2025）的论文。
2.2 Audio Flamingo Next 训练
2.2.1 数据整理
作为数据整理的第一步，我们识别了 Audio Flamingo 模型系列中的关键局限性。这些局限包括核心技能执行上的差距（例如，计数和说话人日志等），以及由于训练期间对某些数据类型接触有限（例如，多语言 ASR、复杂多说话人音频理解等）所导致的数据分布差距。为了解决这些不足，我们从两个来源整理训练数据：现有的公开数据集和从开放互联网收集的原始音频，随后我们对后者进行合成标注。我们的最终数据集包含 1.08 亿个样本，总计 100 万小时的音频。我们按照以下维度收集数据：
1. 音乐理解。我们将来自 Music Flamingo 的数据纳入训练混合集，包括来自 MF-Skills 的标题生成和问答数据。此外，我们扩展了音乐到歌词的数据，特别是针对非英语歌曲，以提升对不同文化中歌词的理解能力。
2. 多人对话语音理解。我们整理了用于多人对话语音的自动语音识别（ASR）和问答（QA）数据，以提升模型追踪说话人轮次、解析重叠语音以及基于对话结构进行推理的能力。这类数据在预训练阶段尤为有用，因为它能教会模型基本的轮次转换和说话人感知技能，这些技能是理解包含多个说话人、背景噪音和音乐的真实长音频的基础。在问答方面，我们聚焦于三项核心技能：(i) 说话人识别——模型需根据给定的语句，判断该语句出自按首次出现顺序排列的哪一位说话人；(ii) 打断识别——模型需识别音频中的打断现象；(iii) 目标说话人语音识别——模型需转录指定说话人的语音内容。我们通过此类数据将 AF-Skills 数据集扩充了总计 45K 个训练样本。
3. 真实世界音频的长描述。尽管 AF2 和 AF3 引入了长音频理解与描述，但先前工作使用的大部分数据仅限于大约 5-10 分钟的音频，这些音频通过拼接较短的片段构建，或者主要在训练后阶段作为对齐技术使用。在 AF-Next 中，我们转而将长音频理解作为训练的核心部分，目标是实现对长格式音频的原生理解与描述。为此，我们从开放互联网上整理了超过 20 万个长视频，时长跨度从 5 到 30 分钟。我们使用智能体网络搜索来发现涵盖不同主题和音频条件的网站与频道，并利用可获取的元数据（如上传者信息和观众评论）来指导筛选。对于每个视频，我们为 10 秒的片段生成四种形式的描述：视频描述、音频描述、语音转录文本以及口语副语言描述。然后，我们提示一个大语言模型（提示词 LABEL:fig:prompt_detailed_caption_audio_only）将这些片段级别的标注整合成一段连贯的音频描述。利用相同的信息，我们还合成了问答数据，主要关注“大海捞针”式问答、时间理解问答和子场景问答，遵循 AF3 中引入的 AudioSkills-XL 方法。我们没有为长音频合成其他类型的问答，因为我们发现当前的流程在这些场景下不够稳健，且更容易产生模型幻觉。
4. 用真实世界数据扩展现有技能。AudioSkills-XL 的很大一部分源自学术数据集（如 AudioSet），这限制了其对真实世界音频的鲁棒性。利用上述收集的长格式音频，我们对信息量丰富的 10-30 秒片段进行采样，并生成涵盖现有 AudioSkills-XL 技能集的问答数据。为了识别此类片段，我们通过向大语言模型提示片段描述来评估信息量。包含更多不同且重叠声学事件的片段会被赋予更高的信息量分数，并被优先选中。这带来了超过 200 万个额外的样本。
5. 多音频数据。为了支持对多个音频输入进行推理，我们整合了 Kumar 等人（2025a）的数据集，并进一步扩展，用于处理交错音频文本指令跟随任务。我们总共收集了 100 万个训练样本。
6. 多轮对话数据。我们进一步扩展了多轮、多音频对话数据，其中包含的问题不仅需要音频理解能力，还需要信息提取和世界知识。我们总共收集了 3 万个样本。
7. 安全性与指令跟随数据。最后，我们合成了安全性与指令跟随数据，以提升 LALM 在这些方面的能力——这在以往的音频语言模型中很大程度上被忽视了。在安全性方面，我们从真实世界数据中识别出不安全音频，并生成相应的问答对和拒绝式回复，教导模型在何时以及如何恰当地拒绝回答。我们的数据总共包含 38.6 万个样本。
8. 多语言 ASR 与 AST。除了来自 AF3 的英语 ASR 数据外，我们还添加了来自以下数据集的多语言 ASR 数据和 AST 数据：Emilia 数据集（He 等人，2024）、CoVoST（Wang 等人，2020）、MUST（Qin 等人，2025）、Amazon-SIFT（Pandey 等人，2025）、ALI 会议（Yu 等人，2022）、aidatatang（北京数据堂科技有限公司，2018）、aishell（Bu 等人，2017）以及 Granary（Koluguri 等人，2025）。
9. 纯文本数据。除了音频文本数据集外，我们还整合了专注于科学、数学、指令跟随和通用知识领域的纯文本 SFT 数据集，以保持模型的文本推理能力。具体来说，我们采用了 Wang 等人（2025a）提出的数据集。
10. 时间锚定思维链。我们提出了时间音频思维链（Temporal Audio Chain-of-Thought），这是一种新颖的推理框架，它教会模型将其中间推理步骤锚定到音频中的时间戳上。此前关于大音频语言模型（LALM）思维链训练的研究普遍报告称，其提升效果较为有限，尤其是与编码和智能体推理等领域相比。我们推测，其中一个原因在于训练数据的性质。现有的音频思维链数据集，例如 AF-Think，大多局限于短视频片段和相对简单的问答，然后再为其附加推理链。然而，在实际应用中，扩展推理对于需要刻意进行证据聚合的复杂问题最为有用。在音频领域，这类问题通常出现在包含多个、重叠且时间上分散事件的真实世界长录音中。
因此，为了实现这一点，我们创建了 AF-Think-Time，这是一个新颖的问题-答案-思维链三元组数据集。AF-Think-Time 的数据来源于具有挑战性的音频素材，包括预告片、电影解说、悬疑故事以及长篇多人对话，并配以需要扩展时间推理能力的问题。我们将推理锚定到时间，原因有二：（i）时间锚定的思维有助于模型在长而复杂的音频中进行导航和推理；（ii）将中间推理过程与带时间戳的事件相关联，可以提升识别性能（Kumar 等人，2026）。我们通过以下方式构建该数据集：首先，使用类似于 Kumar 等人（2026）的流程为每段音频生成带时间戳的字幕，然后基于这些字幕提示一个大语言模型（LLM）来合成三元组（参见提示词 LABEL:fig:prompt_thinking）。AF-Think-Time 总共包含 43K 个训练样本，思维链的平均长度为 446.3 个词。
2.2.2 训练课程
我们采用四阶段课程训练 AF-Next，每个阶段使用不同的数据混合策略，旨在促进稳健均衡的学习，同时逐步增加上下文长度。我们设计了一个数据加载器，根据每个数据集的预定义混合权重从多个数据集中采样。在每个训练周期中，模型都会接触到该数据集规模的数据。在每个阶段内，我们根据验证性能逐步降低低质量数据的权重，并提高高质量或更具挑战性数据的权重。我们的核心假设是：不同能力会在训练的不同阶段涌现——一些基础技能在早期习得，而更复杂的技能和长上下文能力则需要后期阶段的专门训练。我们在表 LABEL:tab:dataset-details 中提供了完整的数据混合比例，并在第 3 节中描述了训练技术，包括训练超参数。
预训练。我们的预训练包含两个阶段，沿用 AF3 的前两个阶段。在第一阶段，我们仅训练音频适配器，同时保持 AF-Whisper 和大语言模型冻结，目标是将音频表示与语言模型嵌入空间对齐。在第二阶段，我们进一步微调音频编码器和适配器，但仍保持大语言模型冻结。两个阶段主要侧重于识别导向的数据，包括分类、字幕生成和自动语音识别。第一阶段的最大音频长度为 30 秒，第二阶段为 1 分钟，而两个阶段的总上下文长度均限制在 8K 个模型 token。
中期训练。我们的中期训练同样包含两个阶段，重点在于将能力从识别拓展至推理与技能习得。在第一阶段，我们对整个模型进行全参数微调。我们保留了预训练阶段使用的数据集，并额外引入了新整理的数据集以及 AudioSkills-XL。由于针对特定技能的监督信号在短音频上仍最容易规模化，该阶段继续强调高质量短音频问答与基础技能数据，同时将最大音频长度提升至 10 分钟，以容纳 AudioSkills 中的长样本。此阶段的总上下文长度上限为 24K 个模型 token。在第二阶段，我们进一步扩展数据混合，加入新收集的长音频描述与问答数据集。为促进模型对该数据及其分布的学习，第一阶段的数据混合权重被降采样至原始混合权重的一半，而所有长音频数据集均被赋予 1.0 的混合权重。此阶段的最大音频长度为 30 分钟，总上下文长度提升至 128K 个模型 token。在中期训练过程中，我们从当前阶段约一半训练进度处取一个检查点，用于初始化下一阶段并继续训练。由此得到的完整训练模型称为 AF-Next-Captioner。
后训练。以中期训练后得到的模型为起点，我们进行基于 GRPO 的强化学习。所有优化设置均遵循 Ghosh 等人（2025a）的方案。在此阶段，我们重点关注多轮对话、安全性、指令遵循能力，以及来自 AudioSkills-XL 的特定技能数据集，主要聚焦于模型在中期训练后表现出的技能。由此得到的模型称为 AF-Next-Instruct。
思维链训练。最后，我们利用 AF-Think-Time 对模型进行链式推理训练。以 AF-Next-Instruct 为起点，我们首先在 AF-Think-Time 上进行 SFT，然后使用后训练阶段的数据混合进行 GRPO 训练。由此阶段得到的模型称为 AF-Next-Think。
2.3 长上下文训练流程
在长音频序列（最长可达数分钟）上训练音频语言模型会带来两个重大挑战：1）音频 token 扩展导致最大序列长度超出标准上下文窗口（例如 32k），2）自注意力机制的二次方内存占用使得标准上下文长度扩展（例如 128k）不可行。我们通过在数据加载器中进行序列级打包以及跨 GPU 的混合序列并行（SP）来解决这两个问题。
| 数据集 | 先前 SOTA | 指标 | 结果 |
| MMAU-v05.15.25（测试集） 声音 | 音乐 | 语音 | 平均 | Audio Flamingo 3 | ACC | 75.83 | 74.47 | 66.97 | 72.42 |
| AF-Next-Instruct | 78.80 | 74.23 | 69.57 | 74.20 | ||
| AF-Next-Think | 78.70 | 74.73 | 71.5 | 75.01 | ||
| AF-Next-Captioner | 79.87 | 75.3 | 72.13 | 75.76 | ||
| MMAR | Audio Flamingo 3 | ACC | 58.5 |
| AF-Next-Instruct | 59.7 | ||
| AF-Next-Think | 61.0 | ||
| AF-Next-Captioner | 63.0 | ||
| MMSU | Gemini-2.5-Flash | ACC | 66.1 |
| AF-Next-Instruct | 59.4 | ||
| AF-Next-Think | 61.2 | ||
| AF-Next-Captioner | 63.3 | ||
| MMAU-Pro | Gemini-2.5-Pro | ACC | 57.4 |
| AF-Next-Instruct | 56.9 | ||
| AF-Next-Think | 58.7 | ||
| 音频描述 Clotho-v2 | AudioCaps | Audio Flamingo 3 | Audio Flamingo 3 | CIDEr ↑ | 0.50 | 0.70 |
| AF-Next-Instruct | 0.52 | 0.74 | ||
| 音频蕴含 Clotho | AudioCaps | Audio Flamingo 3 | Audio Flamingo 3 | ACC | 93.3 | 95.0 |
| AF-Next-Instruct | 94.2 | 96.0 | ||
| NonSpeech7k | Audio Flamingo 3 | ACC | 85.7 |
| AF-Next-Instruct | 86.2 | ||
| CMM 幻觉 | Audio Flamingo 3 | ACC | 86.5 |
| AF-Next-Instruct | 87.0 | ||
| CompA-R-test | Audio Flamingo 3 | ACC | 98.0 |
| AF-Next-Instruct | 98.7 | ||
| LibriSQA | Audio Flamingo 3 | GPT4o | 8.7 |
| AF-Next-Instruct | 9.3 | ||
| NSynth 音源 | 乐器 | Pengi | Qwen-A | ACC ↑ | 62.0 | 78.8 |
| AF-Next-Instruct | 66.7 | 81.7 | ||
| Medley-Solos-DB 乐器 | Audio Flamingo 2 | ACC | 85.80 |
| AF-Next-Instruct | 92.13 | ||
| MuchoMusic | Music Flamingo | ACC | 74.5 |
| AF-Next-Instruct | 75.6 | ||
| SongCaps GPT5-覆盖率 | GPT5-正确性 | Audio Flamingo 3 | GPT5 | 6.7 | 6.2 |
| AF-Next-Instruct | 8.8 | 8.9 | ||
| LongAudioBench | Gemini-2.5-Pro | GPT4o | 60.4 |
| Audio Flamingo 3 | 68.6 | ||
| AF-Next-Instruct | 73.9 | ||
| +语音 | Gemini-2.5-Pro | GPT4o | 66.2 |
| Audio Flamingo 3 | 72.9 | ||
| AF-Next-Instruct | 81.2 | ||
| LibriSpeech（英文） test-clean | test-other | Phi-4-mm | Qwen2.5-O | WER | 1.67 | 3.4 |
| Audio Flamingo 3 | 1.57 | 3.13 | ||
| AF-Next-Instruct | 1.54 | 2.76 | ||
| SPGISpeech（英文） | Qwen2-A-Inst | 词错误率（WER） | 3.0 |
| Audio Flamingo 3 | 1.86 | ||
| AF-Next-Instruct | 1.91 | ||
| TEDLIUM（英文） | Phi-4-mm | 词错误率（WER） | 2.9 |
| Audio Flamingo 3 | 3.5 | ||
| AF-Next-Instruct | 3.3 | ||
| GigaSpeech（英文） | Phi-4-mm | 词错误率（WER） | 9.8 |
| Audio Flamingo 3 | 10.2 | ||
| AF-Next-Instruct | 9.8 | ||
| Common Voice 15（英文） | Phi-4-mm | 词错误率（WER） | 7.6 |
| Audio Flamingo 3 | 7.4 | ||
| AF-Next-Instruct | 7.2 | ||
| VoxPopuli（英文） | Phi-4-mm | 词错误率（WER） | 5.9 |
| Audio Flamingo 3 | 5.6 | ||
| AF-Next-Instruct | 5.4 |
| 任务 | 模型 | AdvBench | AlpacaEval | CommonEval | OpenBookQA |
|---|---|---|---|---|---|
| VoiceBench | Qwen2.5-O | 99.62 | 4.33 | 3.84 | 79.12 |
| Audio Flamingo 3 | 98.26 | 4.19 | 3.40 | 66.81 | |
| AF-Next-Instruct | 98.84 | 4.43 | 3.96 | 80.9 | |
| CoVoST2（语音翻译，BLEU） | |||||
| 语言 | 模型 | 中文 | 日语 | 阿拉伯语 | 德语 |
| 英语 X | Phi-4-mm | 38.0 | 31.9 | 9.9 | 35.3 |
| AF-Next-Instruct | 38.2 | 29.6 | 21.9 | 31.4 | |
| X 英语 | Phi-4-mm | 24.9 | 33.3 | 5.5 | 37.9 |
| AF-Next-instruct | 25.6 | 27.2 | 29.4 | 33.0 | |
序列打包。我们采用三阶段打包策略来处理异构序列长度：（i）SP 感知采样，分布式采样器在数据并行（DP）组之间划分数据，同时确保 SP 组内的所有 GPU 加载相同的样本索引。当 SP 度为 时，有效 DP 副本数减少为 。来自每个 SP 秩的索引交错排列，使得组内的每个秩在每一步都加载相同的批次。批次级混洗提供了随机性，同时不破坏这种对齐；（ii）填充与截断，数据整理器将一个批次中的所有序列填充到最长序列与最大上下文长度中较短的那个长度，在非填充位置上构建二元注意力掩码，并用忽略索引填充标签；（iii）音频 token 扩展，在音频编码阶段，每个音频占位符 token 被替换为可变数量的音频嵌入 token，具体数量由基于片段时长的嵌入掩码决定。
混合序列并行。我们使用统一序列并行（USP）将注意力分布到多个 GPU 上，该并行分解为 Ulysses 度（基于 all-to-all）和 Ring 度（基于点对点）。系统为每种方式构建独立的 NCCL 进程组：一个 Ulysses 组、一个 Ring 组和一个数据并行组。Ulysses 注意力（Jacobs 等人，2023）通过 all-to-all 集合通信在 GPU 之间重新分配序列维度和头维度，使每个 GPU 拥有完整序列但仅拥有部分注意力头（如图 3 所示）——在高带宽互连内高效，但在跨节点时成本高昂。Ring 注意力（Liu 等人，2023）则通过点对点传输在环形拓扑中循环 KV 块，可跨节点扩展，但会引入与环大小成正比的顺序延迟。混合 SP 将两者结合：Ulysses 在 all-to-all 带宽充足的节点内运行，而 Ring 跨节点运行，从而在两个层面都保持通信高效（Fang 和 Zhao，2024）。
3. 实验
实验设置。我们在 128 块 NVIDIA H100 GPU 上对 AF-Next 进行预训练、中期训练、后训练和 CoT 训练。关于每个训练阶段的批次大小、学习率和优化器的更多细节见附录 LABEL:sec.afnext_training_details。为了评估 AF-Next 描述器，我们使用该模型为音频生成描述，并以纯文本模式向 GPT-5.2 输入该描述及相关问题。
基线。我们将所有 3 个模型变体与近期最先进的 LALM 进行了评估对比，包括 GAMA（Ghosh 等人，2024）、Audio Flamingo（Kong 等人，2024）、Audio Flamingo 2、Audio Flamingo 3、Qwen-A（udio）（Chu 等人，2023b）、Qwen2-A（udio）（Chu 等人，2024）、Qwen2-A（udio）-（Inst）ruct、Qwen2.5-O（mni）（Xu 等人，2025a）、Qwen3-O（mni）（Xu 等人，2025b）、R1-AQA（Li 等人，2025a）、Pengi（Deshmukh 等人，2023）、Phi-4-mm（Abouelenin 等人，2025）、百川音频（Li 等人，2025b）、Step-Audio-Chat（Huang 等人，2025）、LTU（Gong 等人，2023b）、LTU-AS（Gong 等人，2023a）、SALMONN（Tang 等人，2023）、AudioGPT（Huang 等人，2023）以及 Gemini（2.0 Flash、1.5 Pro、2.5 Flash 和 2.5 Pro）（Team 等人，2023）（注意：由于速率限制较低，我们未在 ASR 基准上评估 Gemini），还有 GPT-4o-audio（Hurst 等人，2024）。对于 LongAudioBench，针对不支持较长音频的模型，我们遵循 Ghosh 等人（2025b）提出的级联方法进行评估。我们运行了所有上述基线，并报告了复现的分数。
评估数据集。我们在多种任务和基准上评估了 AF-Next 系列模型，包括音频分类（NSynth（音源与乐器）（Engel 等人，2017）、NonSpeech7k（Rashid 等人，2023）、LibriSQA（Zhao 等人，2023））、侧重推理的音频问答（MMAU（Sakshi 等人，2024）（v05.15.25）、MMAU-Pro（Kumar 等人，2025b）、MuchoMusic（感知版本）（Zang 等人，2025；Weck 等人，2024）、MMAR（Ma 等人，2025）、MMSU（Wang 等人，2025b）、CompA-R-test（Ghosh 等人））、多模态幻觉检测（CMM（Leng 等人，2024））、自动语音识别（Librispeech（干净与其他）（Panayotov 等人，2015）、SPGISpeech（O’Neill 等人，2021）、TEDLIUM（Rousseau 等人，2012；Hernandez 等人，2018）和 Voxpopuli（Wang 等人，2021））、LongAudioBench（Ghosh 等人，2025b）以及 SongCaps（Ghosh 等人，2025a）。为计算准确率，我们采用与真实标签的精确字符串匹配，或遵循（Deshmukh 等人，2023）的方法，使用基于 CLAP 的检索，该检索通过开源 AF-CLAP（Ghosh 等人，2025b）实现。对于多选题，AF-Next 通常仅输出所选选项。当模型给出更冗长或开放式的回答时（例如 AF-Next-Think），我们应用多个正则表达式模式来提取所选选项。尽管 AF-Next 支持更广泛的能力，包括多人语音识别、说话人日志、带时间戳的字幕生成以及语音到语音交互等，但我们将本次提交限定在最广泛使用的基准上，并将这些额外任务的评估留待未来工作。
4. 结果
在表 1 中，我们展示了 Audio Flamingo Next 在一系列多样化的音频理解、推理和语音识别基准上的全面评估。AF-Next-Instruct 确立了其作为最强完全开源 LALM 的地位，显著优于先前的开源模型，并且在大多数任务上与最先进的开源权重和闭源模型保持高度竞争力，同时常常超越它们。此外，我们的 AF-Next-Think 和 AF-Next-Captioner 变体带来了持续的性能提升，进一步推动了性能边界。我们在项目网站上提供了定性示例。
音频理解与推理。在 MMAU-v05.15.25 基准上，AF-Next-Instruct 取得了 74.20 的平均准确率，超越了 Audio Flamingo 3（72.42）。AF-Next-Think 进一步将这一成绩提升至 75.01，而引入字幕生成器流水线后的 AF-Next-Captioner 则取得了 75.76 的最佳结果，在全部三个子类别中均有提升：声音（79.87）、音乐（75.3）和语音（72.13）。在 MMAR 基准上呈现出类似趋势，AF-Next-Instruct（59.7）已优于 AF3（58.5），而我们的 AF-Next-Captioner 变体将准确率推高至 63.0——相比 AF3 实现了 4.5 个百分点的绝对提升。在 MMSU 基准上，尽管闭源模型 Gemini-2.5-Flash 以 66.1 领先，但 AF-Next 大幅缩小了差距：我们的 AF-Next-Captioner 变体达到了 63.3，而指令微调变体为 59.4。在更具挑战性的 MMAU-Pro 基准上，AF-Next-Instruct（56.9）超越了闭源模型 Gemini-2.5-Pro（57.4），而 AF-Next-Think 则将这一领先优势扩大到 58.7。这些结果表明，测试时计算策略能提供互补性优势：链式推理有助于处理需要多步推理的任务，而字幕生成器增强在需要更丰富的声学描述来支撑模型推理时尤为有效。
音频字幕生成、蕴含关系判断与分类。在音频字幕生成质量上，AF-Next-Instruct 在 Clotho-v2（CIDEr：0.52 对比 0.50）和 AudioCaps（0.74 对比 0.70）上均优于 AF3。在音频蕴含关系判断方面，它在 Clotho 上达到 94.2，在 AudioCaps 上达到 96.0，分别超越了 AF3 原本已很出色的 93.3 和 95.0 成绩。在 NonSpeech7k 基准的声音事件分类任务上，AF-Next 达到了 86.2 的准确率（AF3 为 85.7），在 CMM 模型幻觉基准上得分为 87.0（AF3 为 86.5），表明其对音频幻觉内容的鲁棒性有所提升。在 CompA-R 基准上，AF-Next 取得了 98.7 的准确率，在 LibriSQA 上获得了 9.3 的 GPT4o 评分，这两项均优于 AF3。
音乐理解。AF-Next 在音乐基准测试中展现出尤为强劲的性能。在 NSynth 上，其音源分类准确率达 66.7，乐器分类准确率达 81.7，大幅超越此前最优的开源模型（Pengi，62.0）和开放权重模型（Qwen-Audio，78.8）。在 Medley-Solos-DB 乐器识别任务中，AF-Next 达到 92.13，较 Audio Flamingo 2 的 85.80 有显著提升。在 MuchoMusic 上，其得分为 75.6，而 Music Flamingo 为 74.5。在 SongCaps 的音乐描述任务中，AF-Next 的 GPT5 覆盖率和正确率得分分别达到 8.8 和 8.9，相比 AF3 的 6.7 和 6.2 实现了大幅改进。
长音频理解。在 LongAudioBench 上，AF-Next-Instruct 以 73.9 的得分大幅领先 AF3（68.6）和闭源模型 Gemini 2.5 Pro（60.4）。在包含语音的变体（+Speech）中，差距更为显著：AF-Next 达到 81.2，而 AF3 为 72.9，Gemini 2.5 Pro 为 66.2。这些结果凸显了 AF-Next 在长上下文音频和语音推理方面的优势。
自动语音识别。AF-Next-Instruct 在多个英文基准测试中取得了具有竞争力或最优的 ASR 性能。在 LibriSpeech 上，它在 LALM 中创下新低，test-clean 词错误率为 1.54，test-other 为 2.76，优于 AF3 以及 Phi-4-mm 和 Qwen2.5-Omni 等开放权重模型。它还在 Common Voice 15（7.2）、GigaSpeech（9.8）和 VoxPopuli（5.4）上取得了最佳词错误率，同时在 SPGISpeech（1.91，AF3 为 1.86）和 TEDLIUM（3.3，Phi-4-mm 为 2.9）上保持竞争力。
语音理解与语音翻译。我们进一步在 VoiceBench 和语音翻译任务上评估了 AF-Next-Instruct，结果见表 2。在 VoiceBench 上，AF-Next-Instruct 在 AlpacaEval（4.43）、CommonEval（3.96）和 OpenBookQA（80.9）上取得了最高分，在这些子任务上均优于开放权重的 Qwen2.5-Omni 和开源的 AF3。值得注意的是，在 OpenBookQA 上，AF-Next 比 AF3 高出 14 分以上，并略微领先 Qwen2.5-Omni（79.12），同时保持了 98.84 的强劲 AdvBench 安全得分。在 CoVoST2 语音翻译上，AF-Next 展现出了与 Phi-4-mm 相竞争的多语言能力。在 ENX 翻译中，AF-Next 在中文（38.2）和阿拉伯语（21.9）上取得了最佳 BLEU 分数——后者相比 Phi-4-mm（9.9）提升了 12 分之多——同时在日语和德语上保持了竞争力。XEN 翻译也呈现出类似模式，AF-Next 在中文（25.6）和阿拉伯语（29.4）上领先，其中阿拉伯语结果再次显示出相比 Phi-4-mm（5.5）的巨大提升。这些结果表明，AF-Next 的多语言语音理解能力在阿拉伯语等代表性不足的语言对中尤为突出，同时在资源更丰富的语言上也保持了有竞争力的表现。
5. 结论
在本文中，我们提出了 Audio Flamingo Next（AF-Next），这是迄今为止 Audio Flamingo 系列中最强大的模型。除了在广泛的当代音频理解基准上取得 SOTA 性能外，AF-Next 还展现出对真实世界用例显著更强的鲁棒性，并支持一系列广泛的能力，包括理解长达 30 分钟的长音频、多轮对话、带时间戳的字幕生成以及多语言 ASR。我们开源了训练代码、模型检查点和核心技术，以支持未来在开放音频语言建模领域的研究。此外，我们引入了时序音频思维链（Temporal Audio Chain-of-Thought），这是一种用于长音频问答的新推理范式，它明确地将中间证据锚定在时间上，从而实现更可靠、更鲁棒的推理。
局限性
AF-Next 存在若干重要局限性。首先，尽管我们大幅扩展了训练数据规模，使其超越此前开放音频语言模型，但互联网规模的音频数据仍然存在噪声，且在不同领域、语言和声学条件下的分布不均衡。特别是低资源语言、罕见声音事件以及专业化的真实世界领域，其代表性仍然不足。未来工作应着力提升开放音频数据集的多样性、平衡性和覆盖范围。
其次，虽然 AF-Next 改进了长音频理解能力，并支持最长 30 分钟的音频输入，但当证据在时间上相隔较远、分布稀疏或分散在多个片段中时，对长上下文的稳健推理仍然具有挑战性。尽管时序音频思维链（Temporal Audio Chain-of-Thought）改善了时间定位能力，但更强的长上下文记忆、检索和证据聚合能力仍是未来工作的重要方向。
第三，我们的评估聚焦于最成熟的基准测试，因此尚未完全覆盖 AF-Next 所支持的若干能力，包括多说话人语音识别、说话人日志、带时间戳的标题生成以及语音到语音交互。为这些能力构建更广泛的评估协议是下一步的重要工作。
参考文献
- A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, 等人 (2025) Phi-4-mini 技术报告：通过混合 LoRA 实现紧凑而强大的多模态语言模型。arXiv 预印本 arXiv:2503.01743。§3 引用。
- 北京数据堂科技有限公司 (2018) Aidatatang_200zh：一个免费的中文普通话语音语料库。注：包含 600 位说话人的 200 小时语音数据，采用 CC BY-NC-ND 4.0 许可协议。外部链接：链接。§2.2.1 引用。
- H. Bu, J. Du, X. Na, B. Wu, 和 H. Zheng (2017) AIShell-1：一个开源普通话语音语料库及语音识别基线。载于 Oriental COCOSDA 2017，已投稿。§2.2.1 引用。
- Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, 和 J. Zhou (2024) Qwen2-Audio 技术报告。外部链接：2407.10759。§2.1, §3 引用。
- 褚昱、许嘉、周翔、杨强、张帅、闫子昂、周畅、周靖人（2023a）《Qwen-Audio：通过统一大规模音频-语言模型推进通用音频理解》。arXiv预印本 arXiv:2311.07919。引用于第1节。
- 褚昱、许嘉、周翔、杨强、张帅、闫子昂、周畅、周靖人（2023b）《Qwen-Audio：通过统一大规模音频-语言模型推进通用音频理解》。外部链接：2311.07919。引用于第3节。
- S. Deshmukh、B. Elizalde、R. Singh、H. Wang（2023）《Pengi：面向音频任务的音频语言模型》。外部链接：2305.11834。引用于第3节、第3节。
- J. Engel、C. Resnick、A. Roberts、S. Dieleman、M. Norouzi、D. Eck、K. Simonyan（2017）《基于WaveNet自编码器的音符神经音频合成》。收录于《国际机器学习大会》，第1068–1077页。引用于第3节。
- 方杰、赵帅（2024）《USP：面向长上下文生成式AI的统一序列并行方法》。arXiv预印本 arXiv:2405.07719。引用于第2.3节。
- S. Ghosh、A. Goel、L. Koroshinadze、S. Lee、Z. Kong、J. F. Santos、R. Duraiswami、D. Manocha、W. Ping、M. Shoeybi 等（2025a）《Music Flamingo：在音频语言模型中扩展音乐理解》。arXiv预印本 arXiv:2511.10289。引用于第2.2.2节、第3节。
- S. Ghosh、Z. Kong、S. Kumar、S. Sakshi、J. Kim、W. Ping、R. Valle、D. Manocha、B. Catanzaro（2025b）《Audio Flamingo 2：具备长音频理解与专家推理能力的音频语言模型》。arXiv预印本 arXiv:2503.03983。引用于第3节、第3节。
- S. Ghosh、S. Kumar、Z. Kong、R. Valle、B. Catanzaro、D. Manocha（2025c）《Synthio：利用合成数据增强小规模音频分类数据集》。收录于第十三届国际学习表征大会。外部链接：Link。引用于第1节。
- S. Ghosh、S. Kumar、A. Seth、C. K. R. Evuru、U. Tyagi、Sakshi、O. Nieto、R. Duraiswami、D. Manocha（2024）《GAMA：具备高级音频理解与复杂推理能力的大规模音频语言模型》。外部链接：2406.11768。引用于第3节。
- S. Ghosh、A. Seth、S. Kumar、U. Tyagi、C. K. R. Evuru、S. Ramaneswaran、S. Sakshi、O. Nieto、R. Duraiswami 和 D. Manocha 合著《CompA：弥合音频-语言模型中组合推理的差距》。发表于第十二届国际学习表征会议，引用自 §3。
- A. Goel、S. Ghosh、J. Kim、S. Kumar、Z. Kong、S. Lee、C. H. Yang、R. Duraiswami、D. Manocha、R. Valle 等人（2025 年）合著《Audio Flamingo 3：以完全开放的大型音频语言模型推进音频智能》。arXiv 预印本 arXiv:2507.08128。引用自 §2.1、§2.1、§2.1。
- A. Goel、Z. Kong、R. Valle 和 B. Catanzaro（2024a）合著《Audio Dialogues：用于音频与音乐理解的对话数据集》。arXiv 预印本 arXiv:2404.07616。引用自 §1。
- A. Goel、K. Sapra、M. Le、R. Valle、A. Tao 和 B. Catanzaro（2024b）合著《OMCAT：全上下文感知 Transformer》。外部链接：2410.12109，链接。引用自 §2.1。
- Y. Gong、A. H. Liu、H. Luo、L. Karlinsky 和 J. Glass（2023a）合著《联合音频与语音理解》。外部链接：2309.14405。引用自 §3。
- Y. Gong、H. Luo、A. H. Liu、L. Karlinsky 和 J. Glass（2023b）合著《倾听、思考与理解》。arXiv 预印本 arXiv:2305.10790。引用自 §3。
- H. He、Z. Shang、C. Wang、X. Li、Y. Gu、H. Hua、L. Liu、C. Yang、J. Li、P. Shi、Y. Wang、K. Chen、P. Zhang 和 Z. Wu（2024 年）合著《Emilia：面向大规模语音生成的多语言、多样化语音数据集》。外部链接：2407.05361，链接。引用自 §2.2.1。
- F. Hernandez、V. Nguyen、S. Ghannay、N. Tomashenko 和 Y. Esteve（2018 年）合著《TED-lium 3：用于说话人自适应实验的数据量翻倍与语料库重划分》。发表于《语音与计算机：第 20 届国际会议，SPECOM 2018，德国莱比锡，2018 年 9 月 18–22 日，会议论文集 20》，第 198–208 页。引用自 §3。
- M. Heydari 和 Z. Duan（2021 年）合著《不要回头：一种基于 RNN 和增强粒子滤波的在线节拍跟踪方法》。发表于《ICASSP 2021-2021 年 IEEE 国际声学、语音与信号处理会议（ICASSP）》，第 236–240 页。引用自 §1。
- A. Huang、B. Wu、B. Wang、C. Yan、C. Hu、C. Feng、F. Tian、F. Shen、J. Li、M. Chen 等人（2025）《Step-Audio：智能语音交互中的统一理解与生成》。arXiv 预印本 arXiv:2502.11946。引用自：§3。
- R. Huang、M. Li、D. Yang、J. Shi、X. Chang、Z. Ye、Y. Wu、Z. Hong、J. Huang、J. Liu、Y. Ren、Z. Zhao 和 S. Watanabe（2023）《AudioGPT：理解与生成语音、音乐、声音及说话头像》。外部链接：2304.12995。引用自：§3。
- A. Hurst、A. Lerer、A. P. Goucher、A. Perelman、A. Ramesh、A. Clark、A. Ostrow、A. Welihinda、A. Hayes、A. Radford 等人（2024）《GPT-4o 系统卡》。arXiv 预印本 arXiv:2410.21276。引用自：§3。
- S. A. Jacobs、M. Tanaka、C. Zhang、M. Zhang、S. L. Song、S. Rajbhandari 和 Y. He（2023）《Deepspeed Ulysses：实现超长序列 Transformer 模型训练的系统优化》。arXiv 预印本 arXiv:2309.14509。引用自：§2.3。
- G. Kim 和 M. Seo（2025）《音频对现代视频大语言模型及其基准测试是否重要？》。arXiv 预印本 arXiv:2509.17901。引用自：§1。
- N. R. Koluguri、M. Sekoyan、G. Zelenfroynd、S. Meister、S. Ding、S. Kostandian、H. Huang、N. Karpov、J. Balam、V. Lavrukhin 等人（2025）《Granary：25 种欧洲语言的语音识别与翻译数据集》。arXiv 预印本 arXiv:2505.13404。引用自：§2.2.1。
- Z. Kong、A. Goel、R. Badlani、W. Ping、R. Valle 和 B. Catanzaro（2024）《Audio Flamingo：一种具备少样本学习与对话能力的新型音频语言模型》。arXiv 预印本 arXiv:2402.01831。引用自：§1、§3。
- S. Kumar、S. Ghosh、Y. Lin、Y. Chen、R. Duraiswami 和 D. Manocha（2025a）《PolyAudio：推进大型音频语言模型中的多音频分析与推理》。外部链接：链接。引用自：§2.2.1。
- S. Kumar、Š. Sedláček、V. Lokegaonkar、F. López、W. Yu、N. Anand、H. Ryu、L. Chen、M. Plička、M. Hlaváček、W. F. Ellingwood、S. Udupa、S. Hou、A. Ferner、S. Barahona、C. Bolaños、S. Rahi、L. Herrera-Alarcón、S. Dixit、S. Patil、S. Deshmukh、L. Koroshinadze、Y. Liu、L. P. G. Perera、E. Zanou、T. Stafylakis、J. S. Chung、D. Harwath、C. Zhang、D. Manocha、A. Lozano-Diez、S. Kesiraju、S. Ghosh 与 R. Duraiswami（2025b）《MMAU-Pro：面向音频通用智能全面评估的挑战性综合基准》。外部链接：2508.13992，链接。引用自：§1、§3。
- S. Kumar、P. Seetharaman、K. Chen、O. Nieto、J. Su、Z. Wang、R. Kumar、D. Manocha、N. J. Bryan、Z. Jin 等人（2026）《TAC：带时间戳的音频描述》。arXiv 预印本 arXiv:2602.15766。引用自：§2.2.1。
- S. Leng、Y. Xing、Z. Cheng、Y. Zhou、H. Zhang、X. Li、D. Zhao、S. Lu、C. Miao 与 L. Bing（2024）《多模态的诅咒：评估大型多模态模型在语言、视觉和音频中的幻觉》。arXiv 预印本 arXiv:2410.12787。引用自：§3。
- G. Li、J. Liu、H. Dinkel、Y. Niu、J. Zhang 与 J. Luan（2025a）《强化学习优于监督微调：以音频问答为例》。arXiv 预印本 arXiv:2503.11197。外部链接：链接。引用自：§3。
- T. Li、J. Liu、T. Zhang、Y. Fang、D. Pan、M. Wang、Z. Liang、Z. Li、M. Lin、G. Dong 等人（2025b）《百川音频：端到端语音交互的统一框架》。arXiv 预印本 arXiv:2502.17239。引用自：§3。
- H. Liu、M. Zaharia 与 P. Abbeel（2023）《基于分块 Transformer 的环形注意力实现近无限上下文》。arXiv 预印本 arXiv:2310.01889。引用自：§2.3。
- Z. Ma、Y. Ma、Y. Zhu、C. Yang、Y. Chao、R. Xu、W. Chen、Y. Chen、Z. Chen、J. Cong、K. Li、K. Li、S. Li、X. Li、X. Li、Z. Lian、Y. Liang、M. Liu、Z. Niu、T. Wang、Y. Wang、Y. Wang、Y. Wu、G. Yang、J. Yu、R. Yuan、Z. Zheng、Z. Zhou、H. Zhu、W. Xue、E. Benetos、K. Yu、E. Chng 与 X. Chen（2025）《MMAR：面向语音、音频、音乐及其混合内容深度推理的挑战性基准》。外部链接：2505.13032，链接。引用自：§3。
- P. K. O’Neill、V. Lavrukhin、S. Majumdar、V. Noroozi、Y. Zhang、O. Kuchaiev、J. Balam、Y. Dovzhenko、K. Freyberg、M. D. Shulman 等人（2021）《Spgispeech：用于全格式端到端语音识别的 5000 小时转录金融音频》。arXiv 预印本 arXiv:2104.02014。引用自：§3。
- V. Panayotov、G. Chen、D. Povey 和 S. Khudanpur（2015）《Librispeech：基于公共领域有声读物的 ASR 语料库》。载于 2015 年 IEEE 国际声学、语音与信号处理会议（ICASSP），第 5206–5210 页。引用自：§3。
- P. Pandey、R. V. Swaminathan、K. V. V. Girish、A. Sen、J. Xie、G. P. Strimel 和 A. Schwarz（2025）《SIFT-50M：用于语音指令微调的大规模多语言数据集》。外部链接：2504.09081，链接。引用自：§2.2.1。
- Z. Peng、J. Yu、Y. Chang、Z. Wang、L. Dong、Y. Hao、Y. Tu、C. Yang、W. Wang、S. Xu 等人（2026）《VIBEVOICE-asr 技术报告》。arXiv 预印本 arXiv:2601.18184。引用自：§1。
- H. Qin、T. Xu、T. Li、Z. Chen、T. Feng 和 J. Li（2025）《MUST：首个多光谱无人机单目标跟踪数据集与统一框架》。外部链接：2503.17699，链接。引用自：§2.2.1。
- A. Radford、J. W. Kim、T. Xu、G. Brockman、C. McLeavey 和 I. Sutskever（2022）《通过大规模弱监督实现鲁棒语音识别》。外部链接：2212.04356。引用自：§2.1。
- M. M. Rashid、G. Li 和 C. Du（2023）《Nonspeech7k 数据集：人类非语音声音的分类与分析》。IET 信号处理 17 (6)，第 e12233 页。引用自：§3。
- A. Rousseau、P. Deléglise 和 Y. Esteve（2012）《TED-lium：一个专用的自动语音识别语料库》。载于 LREC，第 125–129 页。引用自：§3。
- S. Sakshi、U. Tyagi、S. Kumar、A. Seth、R. Selvakumar、O. Nieto、R. Duraiswami、S. Ghosh 和 D. Manocha（2024）《MMAU：大规模多任务音频理解与推理基准》。arXiv 预印本 arXiv:2410.19168。引用自：§3。
- C. Tang、W. Yu、G. Sun、X. Chen、T. Tan、W. Li、L. Lu、Z. Ma 和 C. Zhang（2023）《SALMONN：迈向大语言模型的通用听觉能力》。外部链接：2310.13289。引用自：§3。
- G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican 等人 (2023) Gemini：一系列能力极强的多模态模型。arXiv 预印本 arXiv:2312.11805。引用于 §3。
- Q. Team (2024) Qwen2.5：基础模型系列。外部链接：Link 引用于 §2.1。
- B. Wang, C. Lee, N. Lee, S. Lin, W. Dai, Y. Chen, Y. Chen, Z. Yang, Z. Liu, M. Shoeybi, B. Catanzaro 和 W. Ping (2025a) Nemotron-cascade：为通用推理模型扩展级联强化学习。引用于 §2.2.1。
- C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino 和 E. Dupoux (2021) VoxPopuli：用于表示学习、半监督学习和解释的大规模多语种语音语料库。arXiv 预印本 arXiv:2101.00390。引用于 §3。
- C. Wang, A. Wu 和 J. Pino (2020) CoVoST 2 和大规模多语种语音到文本翻译。外部链接：2007.10310, Link 引用于 §2.2.1。
- D. Wang, J. Wu, J. Li, D. Yang, X. Chen, T. Zhang 和 H. Meng (2025b) MMSU：大规模多任务口语理解与推理基准。arXiv 预印本 arXiv:2506.04779。引用于 §3。
- J. Wang, S. Yan, L. Zheng, J. Wu 和 Y. Mao (2025c) 视听世界模型：迈向视觉与听觉的多感官想象。arXiv 预印本 arXiv:2512.00883。引用于 §1。
- B. Weck, I. Manco, E. Benetos, E. Quinton, G. Fazekas 和 D. Bogdanov (2024) MuChoMusic：评估多模态音频语言模型中的音乐理解能力。arXiv 预印本 arXiv:2408.01337。引用于 §3。
- J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang 等人 (2025a) Qwen2.5-Omni 技术报告。arXiv 预印本 arXiv:2503.20215。引用于 §3。
- J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu 等人 (2025b) Qwen3-omni 技术报告。arXiv 预印本 arXiv:2509.17765。引用于 §1, §3。
- F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu 和 H. Bu (2022) M2MeT：ICASSP 2022 多通道多方会议转录挑战赛。外部链接：2110.07393, Link 引用于 §2.2.1。
- Y. Zang、S. O’Brien、T. Berg-Kirkpatrick、J. McAuley 与 Z. Novack（2025）《你真的在听吗？提升音乐问答基准中的感知意识》。arXiv 预印本 arXiv:2504.00369。引用自：第 3 节。
- Z. Zhao、Y. Jiang、H. Liu、Y. Wang 与 Y. Wang（2023）《LibriSQA：通过新颖数据集与框架推进自由形式及开放式口语问答》。arXiv 预印本 arXiv:2308.10390。引用自：第 3 节。
Sreyan Ghosh
Arushi Goel
Kaousheik Jayakumar
Lasha Koroshinadze
Nishit Anand
Zhifeng Kong
Siddharth Gururani
Sang-gil Lee
Jaehyeon Kim
Aya Aljafari
Chao-Han Huck Yang
Sungwon Kim
Ramani Duraiswami
Dinesh Manocha
Mohammad Shoeybi
Bryan Catanzaro
Ming-Yu Liu
Wei Ping
NVIDIA, USA
University of Maryland, USA
Project-Leads. Ordering was decided with a coin toss.
Model
Abstract
We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds, and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio–language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training, and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with, and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code, and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner, meant for QA, advanced reasoning, and detailed captioning, respectively.
1. Introduction
Audio, spanning speech, environmental sounds, and music, is central to how humans perceive and interact with the world. Robust audio understanding enables core capabilities such as conversation, situational awareness, and music listening, and underpins applications including automatic speech recognition (ASR), audio captioning, and music information retrieval (MIR). Historically, these problems were studied in isolation using small, task-specific models (Peng et al., 2026; Heydari and Duan, 2021). More recently, Large Audio Language Models (LALMs) trained at scale have begun to unify these tasks, demonstrating strong transfer and broad coverage across domains (Goel et al., 2024a; Xu et al., 2025b). Yet, compared to vision-language models (VLMs), progress in scaling open LALMs has been noticeably slower, further limiting audio’s role in general-purpose multimodal systems and downstream efforts such as audio generation and world modeling (Wang et al., 2025c; Kim and Seo, 2025; Ghosh et al., 2025c).
A key barrier is that much of open LALM development has been either closed or tightly coupled to a small set of academic benchmarks. While benchmarks are valuable, they encode biases and incomplete coverage (Kumar et al., 2025b), and audio benchmarks in particular are still emerging. As a result, benchmark-centric training can yield models that perform well on curated test sets but generalize poorly to long, noisy, and diverse real-world audio. Recent frontier systems illustrate both the opportunity and the gap: models in the Audio Flamingo (Kong et al., 2024) and Qwen (Chu et al., 2023a) families introduced capabilities such as long-form audio understanding and multi-turn audio dialogue that are not yet comprehensively evaluated by standardized benchmarks, motivating a shift toward data and training recipes that better reflect applications in the real world.
Main Contributions. We present Audio Flamingo Next (AF-Next), a fully open111By fully open, we mean that the model’s weights, training data, and code will be publicly released, with full transparency about the training methodology (unlike open-weights and closed models). Due to the licensing and scope of the training data used in the work, all releases will be under a research-only license. generalist Large Audio-Language Model that achieves state-of-the-art performance across 20+ audio understanding and reasoning benchmarks, while substantially improving robustness to long and complex real-world audio. AF-Next is a first step towards scaling fully open audio understanding beyond academic datasets and benchmarks by leveraging internet-scale audio data and post-training for reasoning. Concretely, we (i) scale training data beyond academic datasets by curating high-quality data from internet-scale sources, with a focus on long, diverse, and acoustically challenging audio that better reflect real deployment conditions; (ii) strengthen and broaden model capabilities across the Audio Flamingo task suite, including improvements in ASR and audio captioning, and the introduction of new capabilities such as multi-talker ASR, timestamped prediction, long-form audio captioning, and instruction following; and (iii) introduce Temporal Audio Chain-of-Thought, a reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio. To support these advances, AF-Next is trained with a four-stage curriculum that includes multiple rounds of supervised fine-tuning and GRPO-based reinforcement learning over carefully curated data mixtures. In summary, our main contributions are:
-
We introduce AF-Next, an open frontier generalist LALM that advances audio understanding and reasoning along multiple axes. AF-Next is, to our knowledge, the first fully open LALM to scale audio understanding to internet-scale data, and extensive experiments across 20+ benchmarks show that it outperforms similarly sized open models by large margins while remaining highly competitive with, and sometimes surpassing, much larger open-weight and closed models, particularly on long and complex real-world audio.
-
We develop a scalable training recipe for next-generation LALMs, spanning internet-scale data curation, targeted capability expansion, and temporally grounded reasoning for long audio. We open-source our training and inference code, and associated techniques to support future research in open LALMs.
-
We open-source three model checkpoints: AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner, designed for general question answering, advanced reasoning, and detailed captioning, respectively.
2. Methodology
2.1 Audio Flamingo Next Architecture
In this section, we describe our proposed architecture for Audio Flamingo Next, also illustrated in Fig. 3. Similar to Audio Flamingo 3 and Music Flamingo, AF-Next has four main components: i) an audio encoder with sliding window feature extraction, ii) an audio projector to project the audio embeddings into the language space of the LLM, iii) a text-only pre-trained LLM backbone, and iv) a streaming TTS. We provide details of each component below.
AF-Whisper Audio Encoder. Following AF3 and Music Flamingo, we adopt the same Whisper-based AF-Whisper audio encoder, further pre-trained on a larger and more diverse corpus, including multilingual speech and multi-talker ASR data. We refer readers to Goel et al. (2025) for the training details of AF-Whisper.
Feature Extraction. Given an audio input , we first resample it to 16 kHz mono and convert the waveform into a 128-channel log mel-spectrogram using a 25 ms window and 10 ms hop size. The spectrogram is then passed through AF-Whisper to obtain hidden representations, denoted by , where . Audio is processed in non-overlapping 30-second chunks. Thus, , the temporal resolution, depends on the audio duration and the maximum number of sliding windows used during training. AF-Whisper outputs features at 50 Hz, after which we apply a stride-2 pooling layer following Chu et al. (2024). The hidden dimension is 1280.
Audio Adaptor. To bridge the audio representations and the LLM text embedding space, we introduce audio adaptor layers, denoted by . Specifically, the AF-Whisper representations are mapped to adapted embeddings , which are then provided to the LLM as audio prompts alongside the textual instruction. We use a 2-layer MLP as our audio adaptor.
Large Language Model. We use Qwen-2.5-7B Team (2024) as the backbone LLM, a decoder-only causal model with 7B parameters, 36 transformer layers, and 16 attention heads. We further extend its context length from 32k to 128k tokens through additional long-context training, described in Section 2.2.2. Similar to Music Flamingo, we replace the original RoPE with Rotary Time Embeddings (RoTE) Goel et al. (2024b), where the rotation angle is defined using each token’s absolute timestamp rather than its discrete index . Concretely, instead of as in standard RoPE, RoTE uses , yielding temporally grounded positional representations. For audio tokens produced at a fixed 40 ms stride (Radford et al., 2022; Goel et al., 2025), we interpolate discrete time positions and feed them into the RoTE module. RoTE is a core component of AF-Next and is particularly important for Temporal Audio Chain-of-Thought, enabling stronger temporal understanding, especially for long-form audio. We plan to release additional AF-Next variants with smaller and larger LLM backbones in future work.
Streaming TTS. To support voice-to-voice interaction, similar to AF3, AF-Next incorporates a streaming TTS module. The module is implemented as a decoder-only transformer that predicts the next audio token conditioned on incoming subword text tokens from the LLM and previously generated audio tokens. For more details, we refer our readers to Goel et al. (2025).
2.2 Audio Flamingo Next Training
2.2.1 Data Curation
As the first step in data curation, we identify the key limitations in the Audio Flamingo family of models. These include gaps in core skill execution (e.g., counting and speaker diarization, etc) as well as distributional gaps caused by limited exposure to certain data types during training (e.g., multilingual ASR, complex multi-speaker audio understanding, etc). To address these shortcomings, we curate training data from two sources: existing publicly released datasets and raw audio collected from the open internet, which we subsequently label synthetically. Our final dataset comprises 108M samples 1M hours of audio. We collect data along the following axes:
1. Music Understanding. We incorporate data from Music Flamingo into the training mixture, including captioning and QA data from MF-Skills. In addition, we expand our music-to-lyrics data, particularly for non-English songs, to improve lyric understanding across diverse cultures.
2. Multi-talker Speech Understanding. We curate ASR and QA data for multi-speaker speech to improve the model’s ability to track speaker turns, resolve overlapping speech, and reason over conversational structure. This data is especially useful during pre-training, as it teaches the model fundamental turn-taking and speaker-sensitive skills that form the basis for understanding real-world long-form audio containing multiple speakers, background noise, and music. For QA, we focus on three core skills: (i) Speaker Identification, where the model is given an utterance and must determine which speaker, ordered by first appearance, produced it; (ii) Interruption Identification, where the model must identify interruptions in the audio; and (iii) Target Speaker ASR, where the model must transcribe speech corresponding to a specified speaker. We expand AF-Skills by a total of 45K training samples with such data.
3. Long Captioning for Real-World Audio. Although AF2 and AF3 introduce long-audio understanding and captioning, most of the data used in prior work was limited to roughly 5-10 minutes of audio, constructed by concatenating shorter clips, or used primarily during post-training as an alignment technique. In AF-Next, we instead make long-audio understanding a core part of training, with the goal of enabling native understanding and captioning of long-form audio. To this end, we curate more than 200K long videos from the open internet, spanning durations of up to 5 to 30 minutes. We use agentic web search to discover websites and channels across diverse topics and audio conditions, and leverage available metadata, such as uploader information and viewer comments, to guide selection. For each video, we generate four forms of captions for 10-second segments: video captions, audio captions, speech transcripts, and spoken-language paralinguistic descriptions. We then prompt an LLM (Prompt LABEL:fig:prompt_detailed_caption_audio_only) to combine these segment-level annotations into a single coherent caption for the audio. Using the same information, we also synthesize QA data, focusing primarily on needle-in-the-haystack QA, temporal understanding QA, and subscene QA, following AudioSkills-XL introduced in AF3. We do not synthesize other QA types for long audio, as we found our current pipeline less robust for those settings and more prone to hallucination.
4. Expanding Existing Skills with Real-World Data. A large portion of AudioSkills-XL is derived from academic datasets such as AudioSet, which limits robustness to real-world audio. Using the long-form audio collected above, we sample informative 10-30 second segments and generate QA data spanning the existing AudioSkills-XL skill set. To identify such segments, we score informativeness by prompting an LLM with the segment caption. Segments containing a higher number of distinct and overlapping acoustic events are assigned higher informativeness scores and are preferentially selected. This leads to 2M+ more samples.
5. Multi-audio Data. To enable reasoning over multiple audio inputs, we incorporate datasets from Kumar et al. (2025a) and further expand them for interleaved audio-text instruction following. In total, we collect 1M training samples.
6. Multi-turn Chat Data. We further expand multi-turn, multi-audio conversational data with questions that require not only audio understanding, but also information extraction and world knowledge. In total, we collect 30K samples.
7. Safety and Instruction-Following Data. Finally, we synthesize safety and instruction-following data to improve these capabilities in LALMs, which have been largely overlooked in prior audio-language models. For safety, we identify unsafe audio from real-world data and generate corresponding QA pairs and refusal-style responses that teach the model when and how to abstain appropriately. Our data consists of a total of 386K samples.
8. Multi-lingual ASR and AST. Along with English-ASR data from AF3, we add multilingual ASR data and AST data from Emilia dataset (He et al., 2024), CoVoST (Wang et al., 2020), MUST (Qin et al., 2025), Amazon-SIFT (Pandey et al., 2025), ALI meeting (Yu et al., 2022), aidatatang (Beijing DataTang Technology Co., Ltd, 2018), aishell (Bu et al., 2017), and Granary (Koluguri et al., 2025).
9. Text-only Data. In addition to audio-text datasets, we also incorporate text-only SFT datasets focusing on science, math, instruction following, and general knowledge domains to maintain the text-reasoning abilities of the model. Specifically, we employ the dataset proposed in Wang et al. (2025a).
10. Time-Grounded CoT. We introduce Temporal Audio Chain-of-Thought, a novel reasoning framework that teaches the model to ground its intermediate reasoning steps to timestamps in the audio. Prior work on CoT training for LALMs has generally reported only modest gains, especially compared to domains such as coding and agentic reasoning. We hypothesize that one reason is the nature of the training data. Existing audio CoT datasets, such as AF-Think, are largely limited to short clips and relatively simple QA, to which reasoning chains are then attached. In practice, however, extended reasoning is most useful for complex problems that require deliberate evidence aggregation. In the audio domain, such problems typically arise in long, real-world recordings with multiple, overlapping, and temporally dispersed events.
Thus, to enable this, we create AF-Think-Time, a novel dataset of question–answer–thinking-chain triplets. AF-Think-Time is curated from challenging audio sources, including trailers, movie recaps, mystery stories, and long-form multi-party conversations, and is paired with questions that demand extended temporal reasoning. We ground reasoning to time for two reasons: (i) temporally grounded thoughts help the model navigate, and reason over long, complex audio, and (ii) conditioning intermediate reasoning on timestamped events can improve recognition performance (Kumar et al., 2026). We construct the dataset by first generating time-stamped captions for each audio using a pipeline similar to Kumar et al. (2026), and then prompting an LLM over these captions to synthesize triplets (see Prompt LABEL:fig:prompt_thinking). AF-Think-Time consists of a total of 43K training samples, with an average of 446.3 words for thinking-chains.
2.2.2 Training Curriculum
We train AF-Next using a four-stage curriculum, where each stage uses a distinct data mixture designed to promote robust and balanced learning while gradually increasing context length. We design a data loader that samples from multiple datasets according to a predefined blending weight for each dataset. In each training epoch, the model is exposed to the size of that dataset. Within each stage, we progressively down-weight lower-quality data and up-weight higher-quality or more challenging data based on validation performance. Our central hypothesis is that different capabilities emerge at different stages of training: some foundational skills are acquired early, whereas more complex skills and long-context abilities require later-stage specialization. We provide the full data mixing ratios in Table LABEL:tab:dataset-details and describe the training technique, including training hyperparameters in Section 3.
Pre-training. Our pre-training consists of two stages, following the first two stages of AF3. In Stage 1, we train only the audio adaptor while keeping both AF-Whisper and the LLM frozen, with the goal of aligning audio representations with the language model embedding space. In Stage 2, we further fine-tune the audio encoder and adaptor while still keeping the LLM frozen. Both stages focus primarily on recognition-oriented data, including classification, captioning, and ASR. The maximum audio length is 30 seconds in Stage 1 and 1 minute in Stage 2, while the total context length in both stages is capped at 8K tokens.
Mid-training. Our mid-training also consists of two stages and focuses on broadening capabilities beyond recognition toward reasoning and skill acquisition. In Stage 1, we perform full fine-tuning of the entire model. We retain the datasets used during pre-training and additionally introduce our newly curated datasets together with AudioSkills-XL. Since skill-specific supervision remains easiest to scale on short audio, this stage continues to emphasize high-quality short-audio QA and foundational skill data, while increasing the maximum audio length to 10 minutes to accommodate long examples from AudioSkills. The total context length in this stage is capped at 24K tokens. In Stage 2, we further expand the mixture with newly collected long-audio captioning and QA datasets. To promote learning of this data and distribution, the Stage 1 mixture is down-sampled to half of its original blend weights, while all long-audio datasets are assigned a blend weight of 1. The maximum audio length in this stage is 30 minutes, and the total context length is increased to 128K tokens. During mid-training, we initialize the next stage from a checkpoint sampled at roughly the halfway point of the current stage and continue training from there. The fully trained model resulting from this process is referred to as AF-Next-Captioner.
Post-training. Starting from the model obtained after mid-training, we perform GRPO-based reinforcement learning. All optimization settings follow Ghosh et al. (2025a). At this stage, we focus on multi-turn chat, safety, instruction following, and selected skill-specific datasets from AudioSkills-XL, primarily focusing on skills where the model shows post mid-training. The resulting model is referred to as AF-Next-Instruct.
CoT-training. Finally, we train the model for chain-of-though reasoning using AF-Think-Time. Starting from AF-Next-Instruct, we first perform SFT on AF-Think-Time, and train with GRPO using the post-training data mixture. The model obtained from this stage is referred to as AF-Next-Think.
2.3 Long-Context Training Pipeline
Training audio language models on long audio sequences (upto several minutes long) introduces two significant challenges: 1) audio token expansion causes the maximum sequence length to exceed standard context windows (e.g., 32k), and 2) the quadratic memory footprint of self-attention makes standard context length extension (e.g., 128k) infeasible. We address both through sequence-level packing in the dataloader and hybrid sequence parallelism (SP) across GPUs.
| Dataset | Prior SOTA | Metrics | Results |
| MMAU-v05.15.25 (test) Sound | Music | Speech | Avg | Audio Flamingo 3 | ACC | 75.83 | 74.47 | 66.97 | 72.42 |
| AF-Next-Instruct | 78.80 | 74.23 | 69.57| 74.20 | ||
| AF-Next-Think | 78.70 | 74.73 | 71.5 | 75.01 | ||
| AF-Next-Captioner | 79.87 | 75.3 | 72.13 | 75.76 | ||
| MMAR | Audio Flamingo 3 | ACC | 58.5 |
| AF-Next-Instruct | 59.7 | ||
| AF-Next-Think | 61.0 | ||
| AF-Next-Captioner | 63.0 | ||
| MMSU | Gemini-2.5-Flash | ACC | 66.1 |
| AF-Next-Instruct | 59.4 | ||
| AF-Next-Think | 61.2 | ||
| AF-Next-Captioner | 63.3 | ||
| MMAU-Pro | Gemini-2.5-Pro | ACC | 57.4 |
| AF-Next-Instruct | 56.9 | ||
| AF-Next-Think | 58.7 | ||
| Audio Captioning Clotho-v2 | AudioCaps | Audio Flamingo 3 | Audio Flamingo 3 | CIDEr ↑ | 0.50 | 0.70 |
| AF-Next-Instruct | 0.52 | 0.74 | ||
| Audio Entailment Clotho | AudioCaps | Audio Flamingo 3 | Audio Flamingo 3 | ACC | 93.3 | 95.0 |
| AF-Next-Instruct | 94.2 | 96.0 | ||
| NonSpeech7k | Audio Flamingo 3 | ACC | 85.7 |
| AF-Next-Instruct | 86.2 | ||
| CMM Hallucination | Audio Flamingo 3 | ACC | 86.5 |
| AF-Next-Instruct | 87.0 | ||
| CompA-R-test | Audio Flamingo 3 | ACC | 98.0 |
| AF-Next-Instruct | 98.7 | ||
| LibriSQA | Audio Flamingo 3 | GPT4o | 8.7 |
| AF-Next-Instruct | 9.3 | ||
| NSynth Source | Instrument | Pengi | Qwen-A | ACC ↑ | 62.0 | 78.8 |
| AF-Next-Instruct | 66.7 | 81.7 | ||
| Medley-Solos-DB Instrument | Audio Flamingo 2 | ACC | 85.80 |
| AF-Next-Instruct | 92.13 | ||
| MuchoMusic | Music Flamingo | ACC | 74.5 |
| AF-Next-Instruct | 75.6 | ||
| SongCaps GPT5-Coverage | GPT5-Correctness | Audio Flamingo 3 | GPT5 | 6.7 | 6.2 |
| AF-Next-Instruct | 8.8 | 8.9 | ||
| LongAudioBench | Gemini-2.5-Pro | GPT4o | 60.4 |
| Audio Flamingo 3 | 68.6 | ||
| AF-Next-Instruct | 73.9 | ||
| +Speech | Gemini-2.5-Pro | GPT4o | 66.2 |
| Audio Flamingo 3 | 72.9 | ||
| AF-Next-Instruct | 81.2 | ||
| LibriSpeech (en) test-clean | test-other | Phi-4-mm | Qwen2.5-O | WER | 1.67 | 3.4 |
| Audio Flamingo 3 | 1.57 | 3.13 | ||
| AF-Next-Instruct | 1.54 | 2.76 | ||
| SPGISpeech (en) | Qwen2-A-Inst | WER | 3.0 |
| Audio Flamingo 3 | 1.86 | ||
| AF-Next-Instruct | 1.91 | ||
| TEDLIUM (en) | Phi-4-mm | WER | 2.9 |
| Audio Flamingo 3 | 3.5 | ||
| AF-Next-Instruct | 3.3 | ||
| GigaSpeech (en) | Phi-4-mm | WER | 9.8 |
| Audio Flamingo 3 | 10.2 | ||
| AF-Next-Instruct | 9.8 | ||
| Common Voice 15 (en) | Phi-4-mm | WER | 7.6 |
| Audio Flamingo 3 | 7.4 | ||
| AF-Next-Instruct | 7.2 | ||
| VoxPopuli (en) | Phi-4-mm | WER | 5.9 |
| Audio Flamingo 3 | 5.6 | ||
| AF-Next-Instruct | 5.4 |
| Task | Model | AdvBench | AlpacaEval | CommonEval | OpenBookQA |
|---|---|---|---|---|---|
| VoiceBench | Qwen2.5-O | 99.62 | 4.33 | 3.84 | 79.12 |
| Audio Flamingo 3 | 98.26 | 4.19 | 3.40 | 66.81 | |
| AF-Next-Instruct | 98.84 | 4.43 | 3.96 | 80.9 | |
| CoVoST2 (Speech Translation, BLEU ) | |||||
| Lang. | Model | ZH | JA | AR | DE |
| EN X | Phi-4-mm | 38.0 | 31.9 | 9.9 | 35.3 |
| AF-Next-Instruct | 38.2 | 29.6 | 21.9 | 31.4 | |
| X EN | Phi-4-mm | 24.9 | 33.3 | 5.5 | 37.9 |
| AF-Next-instruct | 25.6 | 27.2 | 29.4 | 33.0 | |
Sequence Packing. We employ a three-stage packing strategy to handle heterogeneous sequence lengths: (i) SP-Aware Sampling, where the distributed sampler partitions data across data-parallel (DP) groups while ensuring all GPUs within an SP group receive identical sample indices. With SP degree , the effective DP replica count reduces to . Indices from each SP rank are interleaved so that every rank in a group loads the same batch at each step. A batch-level shuffle provides stochasticity without breaking this alignment; (ii) Padding and Truncation, where the data collator pads all sequences in a batch to the shorter of the longest sequence and the maximum context length, constructs a binary attention mask over non-padding positions, and pads labels with an ignore index; and (iii) Audio Token Expansion, where during audio encoding stage, each audio placeholder token is replaced by a variable number of audio embedding tokens determined by the clip’s duration-based embedding mask.
Hybrid Sequence Parallelism. We distribute attention across GPUs using Unified Sequence Parallelism (USP), decomposed into a Ulysses degree (all-to-all based) and a Ring degree (point-to-point based), with . The system constructs separate NCCL process groups for each: a Ulysses group, a Ring group, and a Data-Parallel group. Ulysses attention (Jacobs et al., 2023) redistributes the sequence and head dimensions across GPUs via all-to-all collectives, giving each GPU the full sequence but only a fraction of the attention heads (as shown in Figure˜3) –efficient within high-bandwidth interconnects but costly across nodes. Ring attention (Liu et al., 2023) instead circulates KV blocks around a ring topology via point-to-point transfers, scaling across nodes but introducing sequential latency proportional to the ring size. Hybrid SP composes both: Ulysses operates within nodes where all-to-all bandwidth is abundant, while Ring spans across nodes, keeping communication efficient at both levels (Fang and Zhao, 2024).
3. Experiments
Experimental Setup. We perform pre-training, mid-training, post-training, and CoT-training of AF-Next on 128 NVIDIA H100 GPUs. Further details on batch size, learning rates, and optimizers for each stage of training are in Appendix LABEL:sec.afnext_training_details. To evaluate AF-Next Captioner, we use the model to generate a caption for the audio and prompt GPT-5.2 in text-only mode with the caption and the associated question.
Baselines. We evaluate all 3 of our model variants against recent SOTA LALMs, including GAMA (Ghosh et al., 2024), Audio Flamingo (Kong et al., 2024), Audio Flamingo 2, Audio Flamingo 3, Qwen-A(udio) (Chu et al., 2023b), Qwen2-A(udio) (Chu et al., 2024), Qwen2-A(udio)-(Inst)ruct, Qwen2.5-O(mni) (Xu et al., 2025a), Qwen3-O(mni) (Xu et al., 2025b), R1-AQA Li et al. (2025a), Pengi (Deshmukh et al., 2023), Phi-4-mm (Abouelenin et al., 2025), Baichun Audio (Li et al., 2025b), Step-Audio-Chat (Huang et al., 2025), LTU (Gong et al., 2023b), LTU-AS (Gong et al., 2023a), SALMONN (Tang et al., 2023), AudioGPT (Huang et al., 2023), and Gemini (2.0 Flash, 1.5 Pro, 2.5 Flash and 2.5 Pro) (Team et al., 2023) (note we do not evaluate Gemini on ASR benchmarks due to low rate limits), as well as GPT-4o-audio (Hurst et al., 2024). For LongAudioBench, for models that do not support longer audio, we follow the cascaded approach for evaluation proposed by Ghosh et al. (2025b). We run all the mentioned baselines, and we report reproduced scores.
Evaluation Datasets. We evaluate our AF-Next series of models on a variety of tasks and benchmarks, including audio classification (NSynth (Source and Instrument) (Engel et al., 2017), NonSpeech7k (Rashid et al., 2023), LibriSQA (Zhao et al., 2023)), reasoning-focused audio QA (MMAU (Sakshi et al., 2024) (v05.15.25), MMAU-Pro (Kumar et al., 2025b), MuchoMusic (perceptual version) (Zang et al., 2025; Weck et al., 2024), MMAR (Ma et al., 2025), MMSU (Wang et al., 2025b), CompA-R-test (Ghosh et al., )), multimodal hallucination detection (CMM (Leng et al., 2024)), ASR (Librispeech (clean and other) (Panayotov et al., 2015), SPGISpeech (O’Neill et al., 2021), TEDLIUM (Rousseau et al., 2012; Hernandez et al., 2018), and Voxpopuli (Wang et al., 2021)), LongAudioBench (Ghosh et al., 2025b) and SongCaps (Ghosh et al., 2025a). To calculate accuracy, we use either exact string matching with the ground truth or CLAP-based retrieval following (Deshmukh et al., 2023), implemented with open-source AF-CLAP (Ghosh et al., 2025b). For MCQ, AF-Next typically outputs only the selected option. In cases where the model provides more verbose or open-ended responses (e.g., AF-Next-Think), we apply multiple regex patterns to extract the chosen option. Although AF-Next supports a broader range of capabilities, including multi-talker ASR, speaker diarization, timestamped captioning, and voice-to-voice interaction, etc, we restrict this submission to the most widely used benchmarks and leave evaluation on these additional tasks to future work.
4. Results
In Table˜1, we present a comprehensive evaluation of Audio Flamingo Next across a diverse suite of audio understanding, reasoning, and speech recognition benchmarks. AF-Next-Instruct establishes itself as the strongest fully open-source LALM, substantially outperforming prior open models and remaining highly competitive with, while often surpassing, state-of-the-art open-weight and closed-source models on the majority of tasks. Furthermore, our AF-Next-Think and AF-Next-Captioner variants yield consistent additional gains, pushing performance even further. We present qualitative examples on our project website.
Audio Understanding and Reasoning. On MMAU-v05.15.25, AF-Next-Instruct achieves an average accuracy of 74.20, surpassing Audio Flamingo 3 (72.42). AF-Next-Think further improves this to 75.01, and incorporating the captioner pipeline AF-Next-Captioner yields the best result of 75.76, with gains across all three subcategories: sound (79.87), music (75.3), and speech (72.13). A similar trend holds on MMAR, where AF-Next-Instruct (59.7) already outperforms AF3 (58.5), and our AF-Next-Captioner variant pushes accuracy to 63.0 – a 4.5-point absolute improvement over AF3. On MMSU, while the closed-source Gemini-2.5-Flash leads at 66.1, AF-Next narrows the gap substantially: our AF-Next-Captioner variant reaches 63.3, compared to 59.4 for the instruct variant. On the more challenging MMAU-Pro benchmark, AF-Next-Instruct (56.9) surpasses the closed-source Gemini-2.5-Pro (57.4), and AF-Next-Think extends this lead to 58.7. These results demonstrate that test-time compute strategies provide complementary benefits: CoT reasoning helps on tasks requiring multi-step inference, while captioner augmentation is particularly effective when richer acoustic descriptions can ground the model’s reasoning.
Audio Captioning, Entailment, and Classification. AF-Next-Instruct improves audio captioning quality on both Clotho-v2 (CIDEr: 0.52 vs. 0.50) and AudioCaps (0.74 vs. 0.70) over AF3. On audio entailment, it achieves 94.2 on Clotho and 96.0 on AudioCaps, improving upon AF3’s already strong results of 93.3 and 95.0, respectively. For sound event classification on NonSpeech7k, AF-Next reaches 86.2 accuracy (vs. 85.7 for AF3), and on the CMM Hallucination benchmark it scores 87.0 (vs. 86.5), indicating improved robustness to hallucinated audio content. On CompA-R, AF-Next achieves 98.7 accuracy, and on LibriSQA it reaches a GPT4o score of 9.3, both improvements over AF3.
Music Understanding. AF-Next demonstrates particularly strong gains on music benchmarks. On NSynth, it achieves 66.7 accuracy for source classification and 81.7 for instrument classification, outperforming the prior best open-source (Pengi, 62.0) and open-weight (Qwen-Audio, 78.8) models by substantial margins. On Medley-Solos-DB instrument recognition, AF-Next reaches 92.13, a notable improvement over Audio Flamingo 2’s 85.80. On MuchoMusic, it scores 75.6 compared to Music Flamingo’s 74.5. For music captioning on SongCaps, AF-Next achieves GPT5 coverage and correctness scores of 8.8 and 8.9, respectively, representing large improvements over AF3’s 6.7 and 6.2.
Long Audio Understanding. On LongAudioBench, AF-Next-Instruct outperforms both AF3 (68.6) and the closed-source Gemini 2.5 Pro (60.4) by a wide margin, achieving 73.9. The gap is even more pronounced on the speech-inclusive variant (+Speech), where AF-Next reaches 81.2 compared to AF3’s 72.9 and Gemini 2.5 Pro’s 66.2. These results highlight AF-Next’s strength in long-context audio and speech reasoning.
Automatic Speech Recognition. AF-Next-Instruct achieves competitive or state-of-the-art ASR performance across multiple English benchmarks. On LibriSpeech, it sets new lows among LALMs with a WER of 1.54 on test-clean and 2.76 on test-other, improving over both AF3 and open-weight models such as Phi-4-mm and Qwen2.5-Omni. It also achieves the best WER on Common Voice 15 (7.2), GigaSpeech (9.8), and VoxPopuli (5.4), while remaining competitive on SPGISpeech (1.91 vs. AF3’s 1.86) and TEDLIUM (3.3 vs. Phi-4-mm’s 2.9).
Voice Understanding and Speech Translation. We further evaluate AF-Next-Instruct on VoiceBench and speech translation tasks in Table˜2. On VoiceBench, AF-Next-Instruct achieves the highest scores on AlpacaEval (4.43), CommonEval (3.96), and OpenBookQA (80.9), outperforming both the open-weight Qwen2.5-Omni and the open-source AF3 across these subtasks. Notably, on OpenBookQA, AF-Next surpasses AF3 by over 14 points and edges out Qwen2.5-Omni (79.12), while maintaining a strong AdvBench safety score of 98.84. On CoVoST2 speech translation, AF-Next demonstrates competitive multilingual capabilities against Phi-4-mm. For ENX translation, AF-Next achieves the best BLEU scores on Chinese (38.2) and Arabic (21.9) — the latter representing a substantial 12-point improvement over Phi-4-mm (9.9) — while remaining competitive on Japanese and German. A similar pattern emerges for XEN translation, where AF-Next leads on Chinese (25.6) and Arabic (29.4), with the Arabic result again showing a dramatic improvement over Phi-4-mm (5.5). These results suggest that AF-Next’s multilingual speech understanding is particularly strong for underrepresented language pairs such as Arabic, while maintaining competitive performance on higher-resource languages.
5. Conclusion
In this paper, we present Audio Flamingo Next (AF-Next), the most capable model in the Audio Flamingo series to date. Beyond achieving SOTA performance on a wide range of contemporary audio understanding benchmarks, AF-Next demonstrates substantially stronger robustness to real-world use cases and supports a broad set of capabilities, including understanding long-form audio of up to 30 minutes, multi-turn chat, timestamped captioning, and multilingual ASR. We open-source our training code, model checkpoints, and core techniques to support future research in open audio-language modeling. In addition, we introduce Temporal Audio Chain-of-Thought, a new reasoning paradigm for long-audio question answering that explicitly grounds intermediate evidence in time, enabling more faithful and robust reasoning.
Limitations
AF-Next has several important limitations. First, although we substantially scale training data beyond prior open audio-language models, internet-scale audio remains noisy and unevenly distributed across domains, languages, and acoustic conditions. In particular, low-resource languages, rare sound events, and specialized real-world domains are still underrepresented. Future work should focus on improving the diversity, balance, and coverage of open audio datasets.
Second, while AF-Next improves long-audio understanding and supports audio up to 30 minutes, robust reasoning over long contexts remains challenging when evidence is temporally distant, sparse, or distributed across multiple segments. Although Temporal Audio Chain-of-Thought improves temporal grounding, stronger long-context memory, retrieval, and evidence aggregation remain important directions for future work.
Third, our evaluation focuses on the most established benchmarks, and therefore does not yet fully cover several capabilities supported by AF-Next, including multi-talker ASR, speaker diarization, timestamped captioning, and voice-to-voice interaction. Building broader evaluation protocols for these capabilities is an important next step.
References
- A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al. (2025) Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743. Cited by: §3.
- Beijing DataTang Technology Co., Ltd (2018) Aidatatang_200zh: a free chinese mandarin speech corpus. Note: 200 hours of speech data from 600 speakers, licensed under CC BY-NC-ND 4.0 External Links: Link Cited by: §2.2.1.
- H. Bu, J. Du, X. Na, B. Wu, and H. Zheng (2017) AIShell-1: an open-source mandarin speech corpus and a speech recognition baseline. In Oriental COCOSDA 2017, pp. Submitted. Cited by: §2.2.1.
- Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou (2024) Qwen2-Audio Technical Report. External Links: 2407.10759 Cited by: §2.1, §3.
- Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou (2023a) Qwen-audio: advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. Cited by: §1.
- Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou (2023b) Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. External Links: 2311.07919 Cited by: §3.
- S. Deshmukh, B. Elizalde, R. Singh, and H. Wang (2023) Pengi: An Audio Language Model for Audio Tasks. External Links: 2305.11834 Cited by: §3, §3.
- J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan (2017) Neural audio synthesis of musical notes with wavenet autoencoders. In International conference on machine learning, pp. 1068–1077. Cited by: §3.
- J. Fang and S. Zhao (2024) Usp: a unified sequence parallelism approach for long context generative ai. arXiv preprint arXiv:2405.07719. Cited by: §2.3.
- S. Ghosh, A. Goel, L. Koroshinadze, S. Lee, Z. Kong, J. F. Santos, R. Duraiswami, D. Manocha, W. Ping, M. Shoeybi, et al. (2025a) Music flamingo: scaling music understanding in audio language models. arXiv preprint arXiv:2511.10289. Cited by: §2.2.2, §3.
- S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro (2025b) Audio flamingo 2: an audio-language model with long-audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983. Cited by: §3, §3.
- S. Ghosh, S. Kumar, Z. Kong, R. Valle, B. Catanzaro, and D. Manocha (2025c) Synthio: augmenting small-scale audio classification datasets with synthetic data. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1.
- S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, Sakshi, O. Nieto, R. Duraiswami, and D. Manocha (2024) GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities. External Links: 2406.11768 Cited by: §3.
- [14] S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. R. Evuru, S. Ramaneswaran, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha CompA: addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference on Learning Representations, Cited by: §3.
- A. Goel, S. Ghosh, J. Kim, S. Kumar, Z. Kong, S. Lee, C. H. Yang, R. Duraiswami, D. Manocha, R. Valle, et al. (2025) Audio flamingo 3: advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128. Cited by: §2.1, §2.1, §2.1.
- A. Goel, Z. Kong, R. Valle, and B. Catanzaro (2024a) Audio dialogues: dialogues dataset for audio and music understanding. arXiv preprint arXiv:2404.07616. Cited by: §1.
- A. Goel, K. Sapra, M. Le, R. Valle, A. Tao, and B. Catanzaro (2024b) OMCAT: Omni Context Aware Transformer. External Links: 2410.12109, Link Cited by: §2.1.
- Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass (2023a) Joint Audio and Speech Understanding. External Links: 2309.14405 Cited by: §3.
- Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass (2023b) Listen, think, and understand. arXiv preprint arXiv:2305.10790. Cited by: §3.
- H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y. Wang, K. Chen, P. Zhang, and Z. Wu (2024) Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation. External Links: 2407.05361, Link Cited by: §2.2.1.
- F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Esteve (2018) TED-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18–22, 2018, Proceedings 20, pp. 198–208. Cited by: §3.
- M. Heydari and Z. Duan (2021) Don’t look back: an online beat tracking method using rnn and enhanced particle filtering. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 236–240. Cited by: §1.
- A. Huang, B. Wu, B. Wang, C. Yan, C. Hu, C. Feng, F. Tian, F. Shen, J. Li, M. Chen, et al. (2025) Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction. arXiv preprint arXiv:2502.11946. Cited by: §3.
- R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu, Y. Ren, Z. Zhao, and S. Watanabe (2023) AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head. External Links: 2304.12995 Cited by: §3.
- A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) GPT-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §3.
- S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y. He (2023) Deepspeed ulysses: system optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509. Cited by: §2.3.
- G. Kim and M. Seo (2025) Does audio matter for modern video-llms and their benchmarks?. arXiv preprint arXiv:2509.17901. Cited by: §1.
- N. R. Koluguri, M. Sekoyan, G. Zelenfroynd, S. Meister, S. Ding, S. Kostandian, H. Huang, N. Karpov, J. Balam, V. Lavrukhin, et al. (2025) Granary: speech recognition and translation dataset in 25 european languages. arXiv preprint arXiv:2505.13404. Cited by: §2.2.1.
- Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro (2024) Audio flamingo: a novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831. Cited by: §1, §3.
- S. Kumar, S. Ghosh, Y. Lin, Y. Chen, R. Duraiswami, and D. Manocha (2025a) PolyAudio: advancing multi-audio analysis & reasoning in large audio language models. External Links: Link Cited by: §2.2.1.
- S. Kumar, Š. Sedláček, V. Lokegaonkar, F. López, W. Yu, N. Anand, H. Ryu, L. Chen, M. Plička, M. Hlaváček, W. F. Ellingwood, S. Udupa, S. Hou, A. Ferner, S. Barahona, C. Bolaños, S. Rahi, L. Herrera-Alarcón, S. Dixit, S. Patil, S. Deshmukh, L. Koroshinadze, Y. Liu, L. P. G. Perera, E. Zanou, T. Stafylakis, J. S. Chung, D. Harwath, C. Zhang, D. Manocha, A. Lozano-Diez, S. Kesiraju, S. Ghosh, and R. Duraiswami (2025b) MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence. External Links: 2508.13992, Link Cited by: §1, §3.
- S. Kumar, P. Seetharaman, K. Chen, O. Nieto, J. Su, Z. Wang, R. Kumar, D. Manocha, N. J. Bryan, Z. Jin, et al. (2026) TAC: timestamped audio captioning. arXiv preprint arXiv:2602.15766. Cited by: §2.2.1.
- S. Leng, Y. Xing, Z. Cheng, Y. Zhou, H. Zhang, X. Li, D. Zhao, S. Lu, C. Miao, and L. Bing (2024) The curse of multi-modalities: evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv preprint arXiv:2410.12787. Cited by: §3.
- G. Li, J. Liu, H. Dinkel, Y. Niu, J. Zhang, and J. Luan (2025a) Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering. arXiv preprint arXiv:2503.11197. External Links: Link Cited by: §3.
- T. Li, J. Liu, T. Zhang, Y. Fang, D. Pan, M. Wang, Z. Liang, Z. Li, M. Lin, G. Dong, et al. (2025b) Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction. arXiv preprint arXiv:2502.17239. Cited by: §3.
- H. Liu, M. Zaharia, and P. Abbeel (2023) Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889. Cited by: §2.3.
- Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y. Chao, R. Xu, W. Chen, Y. Chen, Z. Chen, J. Cong, K. Li, K. Li, S. Li, X. Li, X. Li, Z. Lian, Y. Liang, M. Liu, Z. Niu, T. Wang, Y. Wang, Y. Wang, Y. Wu, G. Yang, J. Yu, R. Yuan, Z. Zheng, Z. Zhou, H. Zhu, W. Xue, E. Benetos, K. Yu, E. Chng, and X. Chen (2025) MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix. External Links: 2505.13032, Link Cited by: §3.
- P. K. O’Neill, V. Lavrukhin, S. Majumdar, V. Noroozi, Y. Zhang, O. Kuchaiev, J. Balam, Y. Dovzhenko, K. Freyberg, M. D. Shulman, et al. (2021) Spgispeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. arXiv preprint arXiv:2104.02014. Cited by: §3.
- V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. Cited by: §3.
- P. Pandey, R. V. Swaminathan, K. V. V. Girish, A. Sen, J. Xie, G. P. Strimel, and A. Schwarz (2025) SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning. External Links: 2504.09081, Link Cited by: §2.2.1.
- Z. Peng, J. Yu, Y. Chang, Z. Wang, L. Dong, Y. Hao, Y. Tu, C. Yang, W. Wang, S. Xu, et al. (2026) VIBEVOICE-asr technical report. arXiv preprint arXiv:2601.18184. Cited by: §1.
- H. Qin, T. Xu, T. Li, Z. Chen, T. Feng, and J. Li (2025) MUST: The First Dataset and Unified Framework for Multispectral UAV Single Object Tracking. External Links: 2503.17699, Link Cited by: §2.2.1.
- A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2022) Robust Speech Recognition via Large-Scale Weak Supervision. External Links: 2212.04356 Cited by: §2.1.
- M. M. Rashid, G. Li, and C. Du (2023) Nonspeech7k dataset: classification and analysis of human non-speech sound. IET Signal Processing 17 (6), pp. e12233. Cited by: §3.
- A. Rousseau, P. Deléglise, and Y. Esteve (2012) TED-lium: an automatic speech recognition dedicated corpus.. In LREC, pp. 125–129. Cited by: §3.
- S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha (2024) MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. arXiv preprint arXiv:2410.19168. Cited by: §3.
- C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang (2023) SALMONN: towards generic hearing abilities for large language models. External Links: 2310.13289 Cited by: §3.
- G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805. Cited by: §3.
- Q. Team (2024) Qwen2.5: a party of foundation models. External Links: Link Cited by: §2.1.
- B. Wang, C. Lee, N. Lee, S. Lin, W. Dai, Y. Chen, Y. Chen, Z. Yang, Z. Liu, M. Shoeybi, B. Catanzaro, and W. Ping (2025a) Nemotron-cascade: scaling cascaded reinforcement learning for general-purpose reasoning models. Cited by: §2.2.1.
- C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux (2021) VoxPopuli: a large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390. Cited by: §3.
- C. Wang, A. Wu, and J. Pino (2020) CoVoST 2 and Massively Multilingual Speech-to-Text Translation. External Links: 2007.10310, Link Cited by: §2.2.1.
- D. Wang, J. Wu, J. Li, D. Yang, X. Chen, T. Zhang, and H. Meng (2025b) MMSU: a massive multi-task spoken language understanding and reasoning benchmark. arXiv preprint arXiv:2506.04779. Cited by: §3.
- J. Wang, S. Yan, L. Zheng, J. Wu, and Y. Mao (2025c) Audio-visual world models: towards multisensory imagination in sight and sound. arXiv preprint arXiv:2512.00883. Cited by: §1.
- B. Weck, I. Manco, E. Benetos, E. Quinton, G. Fazekas, and D. Bogdanov (2024) MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. arXiv preprint arXiv:2408.01337. Cited by: §3.
- J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, et al. (2025a) Qwen2.5-Omni Technical Report. arXiv preprint arXiv:2503.20215. Cited by: §3.
- J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, et al. (2025b) Qwen3-omni technical report. arXiv preprint arXiv:2509.17765. Cited by: §1, §3.
- F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu (2022) M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge. External Links: 2110.07393, Link Cited by: §2.2.1.
- Y. Zang, S. O’Brien, T. Berg-Kirkpatrick, J. McAuley, and Z. Novack (2025) Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks. arXiv preprint arXiv:2504.00369. Cited by: §3.
- Z. Zhao, Y. Jiang, H. Liu, Y. Wang, and Y. Wang (2023) LibriSQA: advancing free-form and open-ended spoken question answering with a novel dataset and framework. arXiv preprint arXiv:2308.10390. Cited by: §3.