Audio-Interaction:统一流式音频交互模型
阅读原文· arxiv.orgAudio-Interaction是一种统一流式音频模型,通过始终在线的感知-决策-回应循环实时聆听声音、环境与指令并即时反应。它基于SoundFlow框架实现端到端数据、训练与部署,包含流式原生数据构建、理解感知训练和异步低延迟推理。配套StreamAudio-2M数据集含260万样本,覆盖7项基本能力、28个子任务;Proactive-Sound-Bench用于评估主动音频干预。在8个基准测试中,Audio-Interaction保持主流音频任务竞争力,同时解锁实时ASR、流式音频指令跟随和主动帮助等离线LALM无法实现的能力。
Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.