[*]同等贡献 \contribution[†]项目负责人 \contribution[🖂]通讯作者 ]京东 \checkdata[项目页面]https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction/ \checkdata[代码仓库]https://github.com/jd-opensource/JoyAI-VL-Interaction \checkdata[发布]模型权重、交互数据及完整系统代码将于2026年6月20日前发布。
JoyAI-VL-Interaction：实时视觉-语言交互智能
姚丁宇
周俊豪
杨晨旭
秦川雨
侯浩文
梁哲铭
王聪聪
曹宇航
叶胜龙
谢帅
谷书焕
黄浩洋
司庆一
段楠
王嘉琪
摘要
现实世界中的许多时刻并不会等待用户主动发问。安全监控画面中燃起火焰，视频通话里闪过一个表情，或者直播中观众心仪的商品转瞬即逝。一旦错过，那个瞬间便不复存在，因为物理世界不会暂停。然而，当今的大模型在设计上大多仍停留在轮次交互模式：它们只在被召唤时才回应，即便是看似具备交互功能的视频通话应用，本质上仍是问答系统，仅在收到轮询或提示词时才做出反应。我们主张一种不同的范式：让模型像人一样存在于世界之中。它持续观察当下正在发生什么，自主决定是开口说话还是保持沉默，进行实时交互，并在遇到难题时委托给后台模型处理。这开启了人机协作的"观察与执行"模式，将AI辅助扩展到更多现实生活场景，并自然契合具身智能对实时性的要求。为了推动交互模型及其跨领域应用，我们做出了两项完全开源的贡献。首先，我们发布了JoyAI-VL-Interaction，一个80亿参数规模、以视觉优先的视觉语言交互模型，其设计天生具备主动性。该模型在内部做出响应决策，每秒选择保持沉默、做出回应或委托给后台模型，并在视觉触发的响应能力和时间感知方面表现出色。我们为其配套提供了一套可迁移的训练方案，由此涌现出我们从未专门训练过的能力，例如引导购物者切换不同的应用界面，或根据幻灯片即兴授课。其次，我们发布了一个围绕该模型构建的完整、可部署系统。该系统将来自摄像头、直播或安全监控的任何实时视频流输入模型，使其真正"在场"于世界之中。它支持数小时的连续视频处理，延迟低于一秒。所有其他组件均可插拔，包括ASR/TTS模块、记忆模块、可视化用户界面，以及一个可连接任何API、模型或智能体的后台大脑。在六个真实世界的流式场景中，人类评估员在质量和时机两方面均以显著优势更偏好JoyAI-VL-Interaction，而非豆包和Gemini的应用内视频通话助手，分别以77.6%和87.9%的胜率击败豆包和Gemini。据我们所知，这是首个开源的、视觉驱动的交互模型，并同时发布了其训练方案、数据和完整可部署系统。借助我们的代码仓库，任何人都可以部署一个能够"看见"当下、自主发言、并在物理世界与数字世界之间流畅切换的助手。我们的目标是推动视觉语言模型超越轮次对话，迈向真正的流式交互，以开放的方式共同前行。
1 引言
最需要人工智能的时刻，往往恰恰是没人有时间去请求它的时刻。一个蹒跚学步的孩子走向滚烫的炉灶；一场比赛中的关键得分在任何人来得及欢呼之前便已结束；独居的年迈父母在隔壁房间摔倒。这些时刻的共同点不仅仅是紧迫性，更是时机：那个需要一句话、一声喝彩或一只扶持之手的瞬间，在任何人想到去请求之前就已到来，又同样迅速地消失。当今的大模型并非为应对此类时刻而构建。大多数模型本质上仍是基于轮次交互的：它们等待被召唤，只有在被提问后才做出回应。这一限制是结构性的，而不仅仅是速度问题。一个基于轮次交互的模型，从构造上就无法感知“当下”：它只在被提示的那一刻才睁开眼睛。而这些场景真正需要的，是一个能持续观察当下正在发生之事，并随着事件展开与世界进行交互的模型。
现有工作并未真正解决这一问题。一方面，实时全模态模型（如 GPT-Realtime-2 [17] 和 Qwen3.5-Omni [22]）是真正端到端的单一模型（音频和视频输入；语音输出），延迟低且无需拼接式流水线。然而，它们的优化目标仍是对话轮次：在用户说完后尽快做出响应。从根本上说，它们仍以对话为核心组织方式。它们等待用户的发言轮次，并非设计用于持续观察视觉流并自行判断何时该主动发言。另一方面，消费级视频通话产品也在探索。豆包应用内的视频通话功能 [2] 通过每隔几秒发起一次后台请求来追求主动性，但其核心仍是基于轮次的，自主性受限于轮询周期：事件只有在下次轮询时才能被观察到，因此系统永远无法在该时间间隔内做出更快的反应。Gemini 应用内的视频通话功能 [7] 则更进一步：它不执行豆包所依赖的后台轮询来监控事件，而是严格保持一问一答的交互模式。第三方面是流式视频理解的研究 [19, 20, 3, 26, 29, 8, 33]，这大多仍停留在实验室阶段。其评估方式通常类似于离线视频理解 [12, 13]，很少能同时满足实际部署所需的各项特性 [31]，即主动响应 [27, 19, 29]、长程记忆 [32, 35, 34] 和实时运行 [28, 25]。这类研究通常一次只推进一个维度。简而言之，这些系统均无法满足实际部署的需求：它们要么是轮次驱动而非事件驱动，要么并未真正存在于实时世界中，要么只活在基准测试里。
JoyAI-VL-Interaction 正是为此而生。我们采用了 Thinking Machines Lab（TML）提出的“交互模型”[11]这一术语，并明确了其标准：交互模型会像人一样，在持续的信息流中自行判断何时才是回应的恰当时机，选择何时发言、何时保持沉默，而非仅在被人点名时才作答。而基于轮次的模型，无论其延迟多低、架构多原生，都无法自主选择回应时机：它只在用户轮次结束时才开口。两者的区别不在于模型回答的速度，而在于它能否自行决定何时值得回应。我们与 TML 几乎同时认识到，交互能力本身值得作为一项独立能力进行规模化扩展；我们的分歧在于实现路径。TML 将语音和视觉融合进一个大模型，并发布了研究预览；而我们则将视觉作为首要驱动因素，将语音作为可插拔的输入/输出模块，使模型足够紧凑以在本地低成本硬件上运行，并开放整个技术栈，以便任何人都能部署、复现并在此基础上进行构建。
贡献一——VL交互模型。我们的核心思路很简单：将"何时行动"变为模型自身按秒学习的决策，并将保持沉默与说话、委派任务一同视为首要行动。我们基于基础视觉语言模型 JoyAI-VL 1.0（§3.1）构建了这种以视觉优先的交互模型。说话与保持沉默是主动性和时间感知的根基——模型需要知道何时值得开口、何时不该开口；委派任务则是模型在问题超出实时推理能力时，借助更强大功能的方式。我们将视觉流的每一秒与其所需的行动配对，生成时间对齐的数据，并大规模构建此类数据（§3.2）；模型由此学会自身的交互能力，将其内化为自身行为，而非依赖外部轮次检测器或活动启发式规则。为了在无限流中保持成本可控，我们使用 AdaCodec[9] 对视频进行编码，该编码器在每一帧可预测的画面上消耗的 token 数量大幅减少，从而在长流中使预算增长显著放缓（§3.1）。我们开源了模型权重、训练方案及其背后的数据。
贡献二——VL交互系统。我们开源了一套开箱即用的完整系统：任何人只需接入摄像头或直播流，交互模型就能实时感知场景动态，与用户进行实时互动，真正融入现场。围绕该模型，系统提供了部署所需的一切：ASR与TTS、可视化界面、长时记忆，以及一个后台桥接层，可连接用户带来的任意API、模型或智能体，全部运行在标准vLLM上，确保在连续数小时的视频中保持实时响应。其中，仅由交互模型自主决定是否交互及何时交互；其余组件仅围绕它进行转译与编排。每个组件都附带开源默认实现，系统提供替换接口，因此部署方可以接入自己的ASR/TTS API或自定义模块，无需重建整个技术栈。该系统运行两个并发循环，通过模型的“委派”动作连接：一个是与用户的实时循环，另一个是异步循环——模型将难题卸载给后台大脑处理，同时保持与用户的在场感，待结果返回后再整合进来。在涵盖六种日常场景的人工对比评估中，JoyAI-VL-Interaction在质量和时机把握上大幅优于豆包和Gemini的应用内视频通话助手，对豆包的胜率为77.6%，对Gemini为87.9%。其最大优势体现在最需要实时响应的场景：在监控与告警方面，它完胜这两个系统；在实时翻译或计数任务中从未落败——这正是那些基于回合制的产品在结构上无法胜任的、事件驱动且需在恰当时机行动的任务（§5）。
这里的价值远不止是打造一个更好的“视频助手”。当一个模型能够实时在场，自主决定观看什么、何时发言，人机协作便从“提交请求并等待”转变为“边看边做”，AI 也能更深入地融入日常生活的方方面面，践行具身智能的核心前提——物理世界无法暂停。同样的能力，也为一系列期待已久的产品奠定了基础：一个真正在场的陪伴者、能翻译或标注所见内容的 AI 眼镜、为视障人士充当眼睛的助手，以及人们尚未想象到的各种应用。我们开源发布了 JoyAI-VL-Interaction 模型、其训练方案、数据以及完整系统，正是为了让社区能够共同构建和探索这一切，乃至更多可能。我们更大的期望是，推动该领域从基于回合的对话，走向真正的流式交互——并且以开源的方式实现。
2 相关工作
2.1 基于回合的模型与产品
当今几乎所有模型都是基于回合的：它们仅在用户发起一个回合时才会行动，被问到时才回应，否则便处于空闲状态。这是一个结构性特征，而非速度问题。一个基于回合的系统，对于世界中发生的、没有伴随用户话语的事件，缺乏做出反应的机制——无论它在被搭话后能多快地回复。近期有两类系统在保持这种基于回合的核心架构的同时，极力提升了响应能力。
实时与全模态模型。
OpenAI 的 GPT-Realtime-2 在单一的语音到语音循环内进行推理 [17]，而阿里巴巴的 Qwen3.5-Omni [22] 是一个原生预训练的全模态模型（文本、音频、图像、视频），具备实时流式传输、内置的轮流发言和打断处理功能，后者已开源发布。这些模型消除了级联架构的延迟，但它们所优化的是对话中的轮流发言：即用户说完后，模型能多快、多自然地做出回应。它们的交互围绕对话组织，并等待用户的回合；它们并非为观察视觉流并自主决定“此刻该发言”而构建。因此，具备开源、实时和多模态特性（如 Qwen3.5-Omni 已经做到的）对于我们设定的场景而言，是必要但不充分的条件。
消费级视频通话产品。
应用内的“视频通话”功能是最为人熟知的实时多模态助手，也是我们最易识别的基准线，但其核心仍然是基于轮次的交互模式。豆包的视频通话功能运行在字节跳动火山引擎的对话式 AI 技术栈上，这是一个可配置的 ASR/VLM/TTS 流水线，具备语音停止/轮次检测功能。其技术文档清晰地说明了机制：当无人说话时，服务器仅缓存传入的视频帧，并不会将其发送给模型，因此应用必须定期触发一个 ExternalTextToLLM 指令才能获取任何分析结果 [24]；此描述基于该技术栈的公开版本，可能与实际部署的产品有所不同。因此，“监控”实际上是一个外挂的时钟，附着在基于轮次的模型之上：对屏幕事件的响应需要等待下一次触发，且永远无法快过轮询间隔。Gemini 的视频通话功能则更为有限；在我们的使用中，它仅能回答提问瞬间可见的那一帧画面。在这两种情况下，何时响应的决策权都不在模型手中。
2.2 交互模型
一条截然不同的、更新的技术路线打破了基于轮次的假设，将交互性内置于模型之中。
TML 交互模型。
思考机器实验室（Thinking Machines Lab）近期将这种模式称为交互模型，并主张——与我们独立且同时得出的结论一致——可扩展的交互性应来自模型本身，而非依赖外部的话轮检测器和活动启发式规则。该模型能实时处理音频、视频和文本，可在无提示的情况下主动发言，根据持续的信息流做出反应，而非等待用户的话轮；同时，与我们的系统类似，它配备了一个异步后台模型，用于处理更慢、更难的推理任务，两者共享上下文。该模型已作为研究预览版发布[11]。两个团队几乎同时得出这一方法，我们将其视为一个信号：从基于话轮的模式向交互式模型的转变，其时机已经成熟。在这种共享的交互-后台设计框架内，我们的方案在两个方面存在有意的差异。在模态方面，我们并未将音频和视觉融合到模型中，而是将语音（ASR/TTS）作为可插拔模块，并将视觉作为模型内在的、主要的模态用于主动交互，以适应“观察并交互”的场景，在该场景中，语音是可互换的输入/输出。在规模方面，TML-Interaction-Small 是一个拥有 276B 参数的混合专家模型（12B 活跃参数），其作者将其描述为在所需延迟下可提供服务的最小模型；而我们则有意选择了一个紧凑的 8B 模型（在交互级响应速度、本地及低成本部署能力与性能之间取得平衡），使其能够在广泛的环境下运行、微调和复现，而不仅限于大型基础设施。
全双工语音。
在语音方面，Kyutai 的 MoshiRAG [4] 是突出的开源实例：一个实时的全双工语音-文本模型，双方可以同时说话和聆听，而非严格交替进行，并且是将交互能力融入模型的最接近的开源先例。其模态和范围（口语对话）使其与我们处理的视觉、事件驱动场景有所不同。
2.3 流式视频理解
一条研究路线让模型能够处理实时到达的视频，而非事后处理，涵盖了流式视频大语言模型、主动响应[27, 19, 29]、实时推理[28, 25]以及长程视频记忆[32, 35, 34]。该路线与我们的模型贡献最为接近，但存在三个差距，使其与可部署的交互模型区分开来。它通常一次只推进一个特性（响应性、主动性或记忆性），而非三者同时推进。它通常在离线基准上进行评估，有时甚至采用离线视频理解的方式，这种方式从未在实时约束条件下测试对现场事件的反应。并且它止步于模型本身，缺乏数小时持续实时存在所需的周边系统（服务、记忆、转换、委派）。
2.4 我们的定位。
沿着两个维度——模型如何交互以及发布了什么——来看，JoyAI-VL-Interaction 占据了一个此前任何工作都未曾涉足的单元格。与原生或产品化的轮次式系统相比，何时行动的决策内置于我们的模型中，每秒都在进行，由事件驱动，因此反应时间受限于推理速度，而非用户的轮次或触发时钟。与 TML 的交互模型相比，我们将视觉作为首要驱动因素，采用“观察并交互”的设置而非音视频对话，并且我们开源的不仅是研究预览版，而是模型本身、其训练配方以及一个完整、可组合、可部署的系统。与 MoshiRAG 相比，我们以视觉为驱动而非以语音为中心，能够对完全不包含对话轮次的事件做出反应。与流式视频研究相比，我们将响应性、主动性和记忆性整合到一个模型中，并将其运行在一个为持续存在而构建的系统内。据我们所知，JoyAI-VL-Interaction 是首个与完整可部署系统一同发布的开源、视觉驱动的交互模型：这是一个所有上述特性交汇而非仅具备单一特性的节点。
3 JoyAI-VL-Interaction 模型
图 1 总结了该模型。JoyAI-VL-Interaction 观看流式视觉输入，并且每秒执行三个动作之一：与用户说话、保持沉默继续观看、或将一个较慢、较困难的任务委托给异步后台模型，该模型的结果在准备好时会被合并回流中。前两个动作与用户形成一个实时循环；第三个动作则与后台形成一个异步循环。
两个设计要点构成了本节其余部分的框架。第一，该模型学习了一种与后台无关的委托协议，因此后台可以替换为任何外部模型、智能体或 API；我们在此处介绍模型端，在 §4 中介绍系统端。第二，该模型本身不处理或生成语音：将语音转为文本以及将文本转回语音的工作交由系统中可插拔的 ASR/TTS 完成，从而使自主的、视觉驱动的核心与可互换的输入/输出解耦。
本节其余部分将基于此行为由外向内构建模型：基础模型与架构（§3.1）、教导何时说话、保持沉默和委托的数据（§3.2），以及训练方案（§3.3），其中涵盖继续训练、训练目标、强化学习和基础设施。
3.1 JoyAI-VL 1.0 与架构
JoyAI-VL 1.0。
JoyAI-VL-Interaction 基于 JoyAI-VL-1.0 构建，其中语言模型从 Qwen3-8B [30] 初始化，视觉编码器为 Qwen3-VL ViT [1]，两者之间的投影层从头开始训练。JoyAI-VL 1.0 通过三个阶段获得：表征对齐、视觉-语言预训练，以及使用在线策略蒸馏 [15] 和强化学习 [14, 23] 的后训练。此时，该模型是一个传统的、基于轮次的视觉语言模型。其每秒的交互行为——决定每秒是说话、保持沉默还是委托——是通过 §3.3 中的交互训练方案后续获得的。
原生流式视频编解码器。
我们使用 AdaCodec [9]（图 2）对视频流进行 token 化。AdaCodec 是一种受预测编码和视频编解码器启发的预测性视觉编码方案：它并非重新编码每一帧，而是仅传输预测无法解释的部分。它仅在参考帧上使用完整的 ViT token，并将中间可预测的帧编码为由运动信息和残差构成的紧凑型 P-token。因此，模型读取的是参考 token 和 P-token 交错组成的流，而非一系列完整的 RGB 帧。当预测代价变得高昂时，预测代价重置机制会开启一个新的参考帧，从而将参考帧精确地放置在场景变化足够大、需要新参考帧的位置。这与我们的场景直接契合：一个始终在线的模型必须消费无界的视频流，而逐帧接口会在每一帧上都消耗完整的 ViT token，因此成本和延迟会随着视频流长度迅速增长。AdaCodec 在每个可预测帧上仅消耗约 16 个 token，仅在场景变化时使用完整的 ViT token，因此预算增长要慢得多，且其高消耗部分随场景变化程度而非帧数扩展。
3.2 面向 VL-Interaction 的数据构建
我们构建的应用共享同一个需求：助手必须在关键时刻自主出现并采取行动，无需任何人提示。安全监控画面中突然出现的火焰，直播中观众想要的商品一闪而过，AI眼镜应在眼前内容出现瞬间进行字幕或翻译，陪伴型设备只在有值得说的事情时才开口。在这些场景中，行动的恰当时机由世界决定，而非用户，一个等待被召唤的模型早已错失良机。本节数据旨在赋予模型这些场景所需的能力：持续观察视觉流，逐秒自主决定是说话、保持沉默，还是将难题交给更强大的后台模型；根据所见内容主动行动，而非等待被询问；同时追踪时间，使其不仅知道发生了什么，还知道何时发生。广泛传授这些能力（而非针对单一用例），是后续所有工作的目标。
具体而言，在每个一秒步长中，模型对已观察到的视觉流采取三种动作之一：保持沉默并继续观看，输出一个 `</silence>` token；说话，输出一个 `</response>` token 及文本回复；或将更慢、更难的子任务委托给异步后台模型。委托遵循一个固定的、与后台无关的协议，该协议基于文本请求和结果定义，因此任何外部模型、智能体或 API 均可作为后台（§4）。我们以一秒为粒度，一遍又一遍地教授这三种决策，覆盖尽可能多的真实生活场景。
数据涵盖的能力。
为覆盖这一范围，数据集包含超过400万个时间对齐的流媒体片段，分为六大类。前五类是：(1) 实时流的主动告警与异常检测；(2) 面向过去、现在和未来时间维度的时间对齐问答；(3) 跨时间的计数与感知；(4) 实时解说与旁白；(5) 多种风格的多轮随意对话，涵盖从以自我为中心短视频的日常交流，到长视频的扩展对话，再到陪伴式交流。第六类(6)则贯穿所有其他类别：委托片段将当前场景与困难的离线子任务配对——包括基于视频的知识问答、STEM问题以及视频推理问题——模型应将这些任务路由到后台处理，而非在线回答。所有类别均采用统一的每秒动作格式，因此每个场景共同教会模型何时发言、何时保持沉默，以及在适当时何时进行委托。
构建时间对齐数据。
每秒级别的监督是使这种行为可学习的关键，且大规模获取成本高昂。每个样本必须在两个维度上准确无误：内容——模型发言时说什么，以及时机——模型应发言、保持沉默或委托的具体秒数。沉默本身是一个一级标签，而非标签缺失，因为任何流中的大多数步骤都应是沉默的，模型必须学会等待，而非每一步都发言。为了在大规模数据上同时保证这两个维度的准确性，我们运行了一个多阶段流水线，配备专门的验证智能体，并有意识地优先考虑质量而非数量。每个候选样本在两个层面接受检查：全局层面，对所有输入帧及完整标注进行检查；局部层面，对标注时间戳对应的帧及其关联回复进行检查。只有通过两项检查的样本才会被收录，一旦任一检查失败即被丢弃。每个数据源按其瓶颈维度进行标注，只有通过所有检查的样本才能进入语料库，从而将异构的原始材料汇聚成统一、清洁的每秒动作格式。
我们针对每个任务族定制了构建方法。对于离线视频问答，开源数据的内容本身已足够可信，因此工作几乎完全集中在时间点上：一个更大的视觉语言模型标记出与每个问题相关的证据出现的时间，并围绕它形成三种时间类型。在回溯型示例中，问题在证据出现之后才提出，模型在问题到达的瞬间做出回答；在即时型示例中，问题恰好在证据出现时提出，模型也在那一刻回答；在前瞻型示例中，问题在证据出现之前提出，模型保持沉默直到证据出现，然后立即回答。对于多轮闲聊，对话的随意性降低了对内容质量的要求，因此我们优先处理时间安排。两个视觉语言模型智能体在持续播放的视频上进行对话，一个根据当前看到的内容提问，另一个观看同一视频流并回答。为了将每一轮对话定位到具体时间点，我们随机采样一个时间点，只给标注模型提供该点附近的三帧画面，这样每个回答都锚定在对应时间戳上，防止时间漂移过大；随后一个验证智能体对每轮对话进行基于视频内容的合理性及质量筛查。
对于评论和旁白，评论者独特的风格本身就是价值所在，因此我们不进行合成，而是收集开源的评论和广播素材，通过自动语音识别还原真实的语音内容，从而保留人类说话和停顿的自然节奏，生成逼真的静音标签。对于计数任务，我们在面向计数的源数据上执行两轮处理：第一轮只保留物体重复出现的视频，第二轮则逐秒标注标签。对于随时间变化的感知任务，我们将时间条件约束插入现成的多轮对话中：例如，在问题中要求回答必须等待精确的时间间隔后才能给出；添加一条指令，要求每过一秒就将累计计数加一；或者限定问题及其回答必须发生的时间窗口。这些规则无需额外人工标注即可生成精确、可验证的时间标注。
对于告警和异常检测而言，时序精度最为关键，因为晚一秒发出的告警所描述的是不同的时刻。因此，我们从两个来源构建此类数据。首先，从开源的时间定位标注中，我们直接将其转换为每秒级别的标签，并对起始点进行一轮验证。其次，对于无标注的网络采集视频，处理流程会先提出一个候选触发窗口，再通过多个阶段对其进行收紧：筛选出可行的候选视频，采样一个窗口，按类型生成目标并进行硬过滤，再应用窗口级验证器；随后，通过密集预检，从查询时刻到候选触发帧之间逐帧扫描（每秒一帧），若目标更早出现则丢弃该样本，从而确保告警标记的是真正的首次出现时刻，之后我们将触发点定位到单个帧。
委托行为的结构剖析。
委托是整个语料库中最复杂、也最具特色的行为：模型在此学会识别实时推理的局限，并在不中断其正在观察的世界的前提下，中途将任务移交给更强大的后台大脑。每个委托行为都是端到端构建的，将触发条件（模型不应在线解决的子任务）与书面请求及返回结果配对。我们通过三种方式合成这些行为：（i）插入现成的纯文本难题，例如STEM问题，此时促使模型转向后台的是问题难度而非模态；（ii）将开源离线视频推理问题转换后，利用大模型将其编织成带标注的多轮对话，使真正困难的视觉问题能在对话过程中自然浮现；（iii）通过多角色智能体流程合成完整行为：规划器编写行为脚本，时间戳/视觉验证器对整个片段及触发帧进行多层级检查，确保触发条件及其时机与屏幕实际内容对应，后台智能体生成深度答案，前台改写器将其转化为自然的回复。
这一集的核心在于它所教授的双循环编排。模型一旦决定委派任务，并不会就此沉默。它首先会向用户发送一条简短的占位回复（例如“让我查一下”），然后生成一个用户永远看不到的隐藏委派 token 和查询，将困难的子任务派发到后台处理，同时视觉流继续滚动。接着，我们注入一段随机延迟，模拟后台可变的推理时间，只有在这段延迟之后，结果才会被重新整合进来，此时模型会在上下文中生成其正式回答。延迟正是关键所在：通过改变延迟时长，我们迫使模型在委派任务尚未完成时保持在场状态，继续观察场景、处理新的交互回合，并在无话可说时保持沉默，而不是僵住直到后台返回结果。这正是用户实时循环与后台异步循环缝合在一起的确切接缝处，模型学会了同时运行这两个循环。跨越全部六个模型家族，无论其原始素材多么不同，每个构建都被简化为相同的每秒标签——沉默、回复和委派——并通过相同的多级验证，这使得如此异质的来源能够训练出一个统一的交互策略。
一种可迁移的方案，以及涌现的迹象。
由于该构建方式是由这种通用的每秒格式所定义，而非由任何单一领域所决定，因此这一方法能够干净地迁移到新的场景中：增加一项能力，只需提供来自该领域的数据流，并以相同的每秒格式进行标注，再通过相同的验证器，而无需改变架构或训练目标。而且，其回报会随着规模扩大而增长。即使只是中等数量的时间对齐数据，也足以涌现出强大的交互行为，这表明该方法具有极高的数据效率，并且时间对齐训练远未达到饱和状态，随着数据量的增长，仍有巨大的提升空间。最引人注目的是，我们从未明确训练过的能力也会自行涌现，包括引导用户在不同应用界面间完成购买，以及根据幻灯片即兴讲授一整堂课。我们认为，这有力地证明了模型正在习得一种通用的“观察与交互”能力，而非记忆特定场景的技巧，并且同样的方法可以催生出下一代始终在线的助手——从能为你所见之物进行解说的AI眼镜，到为视障人士充当眼睛的伴侣和辅助工具。我们目前所看到的，仅仅是个开始。
3.3 VL-Interaction 的训练方法
继续训练。
从 JoyAI-VL 1.0 开始，通过一个单一的监督训练阶段，将基础模型转变为交互模型；随后再进行强化学习。我们将第 3.2 节中的时间对齐交互数据，混入一个由传统轮次数据组成的大型数据池中，并在合并后的语料库上进行微调。这呼应了第 3.2 节的观点：激发交互能力是数据高效的，并且时间对齐训练远未达到饱和。
训练目标。
在时间对齐数据上，静默步骤的数量远远超过模型说话的步骤，因此监督目标主要由 `</silence>` token 主导。在标准 SFT 损失下，这种不平衡会将梯度推向持续静默，并稀释用于响应的信号。因此，遵循我们流式原生训练 [31] 的方法，我们根据角色对助手 token 进行加权。设 `\mathcal{A}` 为监督的助手 token 位置，`\mathcal{C}` 为控制 token 位置，其中每个控制 token 要么是 `</silence>`，要么是 `</response>`。我们为连续静默段中的第一个 `</silence>` 分配权重 `w_{\text{first}}`，为后续的 `</silence>` 分配权重 `w_{\text{cont}}`，为 `</response>` 分配权重 `w_{\text{resp}}`；其他所有位置分配权重 `1`。我们设置 `w_{\text{first}} = 1`，`w_{\text{cont}} = \gamma`（其中 `\gamma < 1`）以降低静默延续的权重，并设置 `w_{\text{resp}} = \beta`（其中 `\beta > 1`）以提高响应开始的权重。委托（delegation）不需要单独的权重，因为它始终位于响应内部。目标函数是归一化的加权交叉熵。
| (1) |
在实际应用中，我们使用 `\gamma = 0.1` 和 `\beta = 5`。我们仅对时间对齐数据应用此加权损失；传统的轮次数据则使用标准 SFT 损失进行训练。
强化学习。
持续训练能够教会这三种行为，但时机的精妙之处很难通过 token 级别的监督来完善：在恰当的时刻开口，在无话可说时保持沉默，以及判断何时将子任务委派出去比直接内联回答更好。我们增加了一个强化学习阶段，采用 GRPO 运行，该阶段直接针对流级别的奖励优化逐秒策略。由于一个长流可能会展开成数百个大部分为静默的轮次，我们通过以答案为中心的窗口采样来保持轨迹的可处理性：对于每个黄金响应，我们构建一条保留流因果性但仅保留对时机至关重要的轮次的轨迹，从而将视野从数百个轮次压缩到少数几个，并将信用集中到时机决策所在之处。奖励机制会奖励那些既正确又在正确窗口内发出的响应、奖励恰当的沉默，以及奖励判断得当的委派；它会惩罚误报（无理由地发言或委派）、时机不当的响应以及退化的始终响应行为。委派行为通过两部分来评分：模型是否将真正困难的子任务（而非那些它应直接内联回答的简单任务）移交出去，以及在后台回复后，模型是否能很好地利用返回的结果，包括在委派尚未完成时保持响应状态。此外，响应内容还会由一个大语言模型评判器根据特定任务的评分标准进行质量评分，从而在简单规则失效时提供一致且细粒度的奖励。
基础设施。
我们在 EasyVideoR1 [21] 上运行我们的强化学习阶段，这是一个用于在视频上训练视觉语言模型的高效强化学习框架。其奖励系统是任务感知的，具有统一的路由和模块化扩展功能，因此针对文本、图像、视频以及我们的流式交互任务的奖励都通过单一管道提供。
4 JoyAI-VL-交互系统
围绕§3中的交互模型，我们构建了一个完整、可部署的系统，其唯一原则是：模型是唯一决定何时说话、何时委派的组件，而其他一切（ASR、TTS、记忆、后台大脑、可视化UI）都是围绕它放置的转导与编排。每个组件都附带一个开箱即用的开源默认实现，并且可以根据部署需求进行替换，包括模型本身——任何使用我们配方（§3.3）训练的VLM都可以替换它。这种“决策在模型中，其余可替换”的划分是该系统的设计主旨，它同时带来了两个特性：系统是可组合的，因此任何人都可以针对自己的领域重建它；同时它今天即可部署，因为外围组件复用了标准基础设施，而非定制化技术栈。
与将语音与视觉一同融合进模型的TML相比，我们将视觉触发——即模型根据所见内容自行决定采取行动——保留为模型原生的内置能力，而将语音视为可互换的输入/输出。因此，将可互换的输入/输出置于模型外部并非弱点，而是有意将自主核心与部署方希望自行选择的部分解耦。
本节其余部分将逐步构建该系统：它运行的两个并发循环、连接这两个循环的后台桥接，以及该桥接如何形成从“看到”到“行动”的闭环（§4.1）；模型周围的可插拔转导、可视化和用户提供的模块（§4.2）；使其在数小时内保持连贯性的长时记忆（§4.3）；以及基于vLLM的原生服务路径和开源发布，以实现亚秒级响应并将系统交到任何人手中（§4.4）。
4.1 两个并发循环：从看到到行动
交互模型同时运行两个对话：一个与用户的实时循环，以及一个与后台大脑的异步循环，两者通过模型的委派动作连接。
数据摄入。
实时循环从数据摄入开始。浏览器客户端通过 WebRTC SDP 协商将摄像头流推送到服务器，或者提交一个由服务器拉取的 RTSP 地址；这两种媒体路径分别由服务器端的 aiortc 和 PyAV 接收并解码，还原为原始帧。一个采样模块以固定间隔对视频流进行降采样，默认频率为 1 Hz，且可根据具体场景配置，以平衡时间细节与实时延迟。采样后的帧被转换为 RGB 格式、JPEG 编码，并封装为 Base64 数据 URL，然后连同活跃的用户查询一起传入一个适配层，该层组装出与 OpenAI 兼容的多模态 Chat Completions 请求，其格式与我们的 vLLM 服务路径所期望的请求相同（§4.4）。
实时循环与异步循环。
这些帧每秒被送入模型，因此模型始终基于当前正在发生的事情而非延迟的批次来采取行动。结合可选的麦克风音频（通过流式 ASR 转录为活跃查询），它们驱动模型：每秒模型执行一个动作——说话、保持沉默或委派任务；当它说话时，流式 TTS 渲染回复，可视化 UI 反映其状态。当它委派任务时，控制权传递给异步循环，在该循环中，后台模型针对后台大脑运行，以处理更慢、要求更高的任务。
后台桥接。
该委托点所指向的边缘是一个后台桥接器，其背后的智能体默认使用用户自身的大模型 API，同时也可自由替换为用户带来的任意智能体（例如 Hermes Agent [16] 和 OpenClaw [18]）。该脚手架原生支持这一机制，并通过文本请求与结果定义了一个固定的、与后台无关的协议（§3.2）。桥接器暴露了一个与后台无关的文本契约：前台发出带标签的查询，桥接器将其规范化为任务 ID、委托问题、前台备注以及有界帧快照。它在固定的超时时间内异步运行一个独立的模型或智能体；超时到期会作为错误事件上报，会话清理时会取消正在进行的任务。结果以“已开始/已就绪/错误”事件的形式返回，完整工件保留在上下文之外，仅将有限摘要编织回交互模型中。由于后台可以是智能体，桥接器还实现了从“观察”到“行动”的闭环：交互模型观察物理世界，并在必要时委托一个任务，由后台智能体在数字世界中执行。这正是 §1 中“观察并执行”前提的具体实现：在一个系统内完成“观察、决策、行动”。
4.2 可插拔组件：转导、可视化与自定义模块
语音识别与语音合成。
在语音方面，我们提供了一个基于开源 ASR [6] 和 TTS 模型 [5] 构建的即用型服务器，用户可以替换为自己的模型或 API，以匹配其语言和偏好的语音。这些模块仅负责模态转换，不参与交互决策。由于语音具有持续时间而模型生成没有，我们预先定义了一条简单规则来保持两者同步：当上一句语音仍在播放时，模型的下一个句子仅以文本形式呈现，不进行合成，从而避免音频在生成后堆积。对于实时解说、分步指导、实时字幕翻译等输出近乎连续的高频场景，我们建议关闭 TTS 运行，并将该机制的更完整处理留待未来工作。
可视化用户界面。
一个内置的 Web 界面（改编自 NVIDIA 的开源项目 live-vlm-webui111https://github.com/nvidia-ai-iot/live-vlm-webui）让运行中的系统变得可见且可控。左侧配置面板可选择视频源——可以是网络摄像头或 RTSP 实时流——并设置采样节奏，即每次推理的帧数和推理间隔；它还提供一种响应模式，可将模型风格从默认的平衡状态调整为更健谈或更冷淡。中央面板显示模型正在观看的实时画面。右侧面板是对话区：它承载着与模型的持续交流，包括之前的轮次，实时报告每次响应延迟和平均延迟，并接受通过键盘输入的文字或通过麦克风输入的语音（ASR）；当模型说话时，其回复会通过 TTS 自动播放。该界面在敏感部署场景中还可作为调试和信任界面，并且可以用自定义前端替换。
自带模块。
除了这些默认功能外，系统还开放给用户提供的模块，以丰富助手的感知或记忆能力。例如，一个人脸识别模块可以让助手记住你是谁，并在特定朋友或家人进入画面时识别出来，从而开启更个性、更深入的互动。我们开放这一切还有一个更重要的原因。轮次式交互是一个早已成熟的领域，经过多年打磨，并围绕它建立了庞大的系统和优化生态；相比之下，视觉语言交互模型才刚刚起步。通过发布完整的技术栈，我们希望在这些已确立的高峰之间为交互模型打开第一个突破口，并邀请社区与我们一同探索这种尚处早期阶段的人机协作方式所能达到的可能性。
4.3 长周期记忆
长时程存在需要一个记忆系统，其占用空间不会随着流式运行数小时而无限制增长，并且其内容构成一个共享上下文，供流式模型和后台大脑共同读取。我们通过基于先前工作[31]构建的可插拔分层记忆来满足这一需求，该记忆按压缩程度递增组织成多个层级。我们将上下文组织成一个三层层级结构，将流划分为：(i) 短期记忆，保留最近几秒的原始视觉 token；(ii) 中期记忆，存储过去短期记忆块的最多文本摘要，覆盖中等压缩程度的数秒内容；以及 (iii) 长期记忆，存储最多经过高度压缩的块，每个块由连续的中期摘要合并而成，因此跨越数秒。除了这些视觉层级之外，还有一个对话记忆，用于保持过去查询和回答在时间上的连贯性。合并步骤在每个边界之前异步运行，因此它们隐藏在主线推理之后，绝不会阻塞实时循环，并且这些层级共同达到大约两小时的上下文长度。由于缓冲区之上的每个层级都以文本形式存储，该记忆还形成了一个稳定的按块划分的前缀，§4.4 的服务路径可以缓存并复用该前缀；它正是为此而设计的，并且保持完全可插拔，可替换为部署所偏好的任何外部记忆。
4.4 服务与运行时
基于 vLLM 的原生服务。
连续数小时的视频会击败那些显而易见的服务策略。在整个历史记录上重新计算注意力会在几百秒内填满上下文窗口；滑动窗口虽然限制了上下文，但破坏了前缀复用，因为连续的步骤不再共享共同的前缀，因此引擎的前缀缓存毫无用处，延迟会超过实时预算；token 剪枝也存在同样的问题。大多数流式视频方法继承了上述某一种缺陷，并且由于构建在普通 Transformer 架构之上，它们无法与 vLLM [10] 和 SGLang [36] 这类高效引擎兼容，这也是它们大多停留在实验室阶段的主要原因。我们则围绕前缀复用设计了内存系统，使得 vLLM 可以直接为其提供服务：第 4.3 节中的文本记忆在每个块中预先填充一次到 KV 缓存中，在随后的每一步中，只计算新观察到的帧和之前的回复，而记忆和块内较早的轮次则被复用而无需重新计算。AdaCodec（第 3.1 节）进一步将每个可预测帧压缩到大约十六个 token，从而保持每一步的计算量很小。以这种方式构建内存正是降低部署门槛的关键：该系统在标准 vLLM 上，能够以亚秒级的端到端延迟，维持超过两小时的连续视频处理。
有状态会话。
运行时围绕服务器端会话进行组织。控制消息、配置更新、会话重置和状态同步，以 JSON 格式通过 WebSocket 或 HTTP 传输，并携带一个 session_id，而服务器则维护每个会话的视频上下文、问答轨迹和内存状态。当组装请求时，这个维护好的上下文会被自动注入到 Chat Completions 请求中，因此单轮客户端-服务器通信只携带即时观测数据，所有长程上下文管理都在后端进行。这也使得上述前缀复用成为可能：注入的上下文正是引擎缓存并复用的、每个块的稳定前缀。
实时鲁棒性。
为了在负载下保持延迟稳定，运行时在连接建立、WebSocket 通道、RTSP 信令以及辅助音视频通道上设置了超时、重连和取消机制，同时通过会话级锁防止请求堆积；必要时，它会丢弃过时的帧，或根据推理序列号将延迟到达的结果回填到正确的交互历史中。适配层根据每个请求的模型标识符将其路由到对应的 VLM 服务，并行执行第 4.3 节所述的中期和长期记忆整合，并返回标准的 JSON 响应。
一切都是开放的。
模型配方、后台桥接、ASR/TTS、记忆、编排、可视化 UI 以及 vLLM 服务路径，都可以在仓库中替换或扩展。同时发布模型配方和完整系统正是关键所在：从单个仓库出发，任何人都可以搭建起属于自己的、自主驱动的现实世界助手。
5 实验
5.1 基准测试
我们评估 JoyAI-VL-Interaction 时，并非使用离线视频理解基准，而是针对本文所关注的、在实时事件驱动场景下用户实际会使用的真实部署产品。具体来说，我们将其与当今最成熟、最具知名度的实时多模态产品——豆包和 Gemini 的应用内视频通话助手（§2.1）——进行直接对比。我们选择了六个日常场景，这些场景完全处于交互模型范式的优势区间：要求在场、在正确时机主动行动、并随时间追踪事件进展，而这正是基于轮次的产品（等待被召唤）在结构上无法胜任的——无论它在被提问后回答得有多快。
场景。
这六个场景分别是：（1）监控与告警，在事件发生的瞬间进行标记；（2）对随时间推移的物体或事件进行实时计数；（3）对屏幕内容进行实时翻译；（4）时间感知，根据自身对经过时间的感知来行动，例如按请求每隔几秒说一次话，或计时某项活动持续了多久；（5）实时解说与引导，在恰当的时刻对正在展开的场景进行叙述或引导用户操作；（6）长时记忆，能够回答关于流媒体中更早之前看到的内容。在这六个场景之下，是定义交互模型的三种能力：实时操作、由所见内容驱动的主动响应，以及长时记忆。因此，这六个场景并非随意设定，而是对这些维度的采样，并且同样的能力远远超出了这个基准测试的范围，可应用于AI眼镜、盲人辅助、安防监控、家用机器人、居家与老年护理、基于视觉的聊天与陪伴应用、体育赛事直播解说，以及将任务委托给后台智能体等场景。所有这些场景中，在场并在正确时机采取行动是整个任务的核心，而这正是基于轮次的产品在结构上天然欠缺的维度，因此这六个场景探究的是范式本身的差距，而非任何单一的产品功能。这六个场景共包含58个案例：监控、计数、翻译和时间感知各10个，解说和记忆各9个。案例主要来自网络上的公开视频素材，少数由我们自行录制。
基线模型。
我们的基线是豆包和 Gemini 应用的视频通话功能，即我们在 §2.1 中单独指出的、核心为回合制的产品，每个产品均按其实际部署给最终用户的方式进行评估。我们使用 2026 年 5 月下旬和 6 月初的当前应用版本，并在匹配条件下用相同的输入驱动每个产品。我们通过自己的系统（§4）运行 JoyAI-VL-Interaction，并分别与每个基线进行成对比较。JoyAI-VL-Interaction 系统采用三级记忆架构，包含 s、 和 （§4.3）。在评估时，我们的 JoyAI-VL-Interaction 系统通过 MediaMTX 将离线视频模拟为 RTSP 实时流源。由于豆包和 Gemini 未暴露系统级 API，我们通过其应用驱动它们：向每个应用播放相同的视频，并在匹配的时间戳提出相同的问题。
协议与指标。
对于每个案例，人工评分员在两个同等重要的维度上对每个系统进行评分：质量，即回答是否正确、相关且表述良好；以及时机，即回答是否在恰当的时刻给出，既不提前也不滞后，以及系统在无话可说时是否保持沉默。每个维度按三个等级评分：好、一般或差，系统在该案例中的得分是其两个维度评分的等权平均值。时机维度是事件驱动场景的核心，也是回合制产品在结构上难以应对的方面（§1）。为减少偏差，系统身份对评分员隐藏，且呈现顺序随机化。共有五名评分员进行评分，均受过大学及以上教育，且为大语言模型领域的研究人员，评分员间一致性较高。对于每个案例，我们随后比较两个系统的加权平均得分，从而得出 JoyAI-VL-Interaction 的胜、平或负。我们分别报告了针对豆包和 Gemini 的每个场景及整体胜率（即 JoyAI-VL-Interaction 获胜的比较次数占比），以及平局率和负率；平局既不算胜也不算负。
| 场景 | JoyAI-VL-Interaction | 平局 | 豆包 |
|---|---|---|---|
| 监控与告警 | 100.0% | 0.0% | 0.0% |
| 实时计数 | 70.0% | 30.0% | 0.0% |
| 实时翻译 | 80.0% | 20.0% | 0.0% |
| 时间感知 | 80.0% | 10.0% | 10.0% |
| 实时解说与引导 | 55.6% | 22.2% | 22.2% |
| 长程记忆 | 77.8% | 22.2% | 0.0% |
| 总体 | 77.6% | 17.2% | 5.2% |
| 场景 | JoyAI-VL-Interaction | 平局 | Gemini |
|---|---|---|---|
| 监控与告警 | 100.0% | 0.0% | 0.0% |
| 实时计数 | 100.0% | 0.0% | 0.0% |
| 实时翻译 | 100.0% | 0.0% | 0.0% |
| 时间感知 | 50.0% | 40.0% | 10.0% |
| 实时解说与指导 | 100.0% | 0.0% | 0.0% |
| 长程记忆 | 77.8% | 22.2% | 0.0% |
| 总体 | 87.9% | 10.3% | 1.7% |
5.2 结果与案例研究
实验结果。
在两项对比中，JoyAI-VL-Interaction 均以大幅优势成为更受青睐的系统，并且在每个场景中，其获胜次数均超过落败次数。与豆包相比，它在 77.6% 的案例中更受青睐，17.2% 为平局，仅 5.2% 落败；与 Gemini 相比，优势更为显著，获胜率达 87.9%，平局 10.3%，落败仅 1.7%。换言之，两个基线系统几乎从未在整体上占据优势。
我们模型最显著的优势恰恰体现在定义这一范式的视觉驱动、时间关键型场景中。在监控与告警——即捕捉事件发生瞬间的最纯粹测试——中，它针对两个基线模型的每一次对比都取得了胜利（100%）。实时翻译与快速计数同样呈现一边倒态势：模型以100%的胜率横扫Gemini，并以80%和70%的胜率完胜豆包，且无一败绩。这些正是"在场并在正确时机行动"构成全部任务核心的场景，也是我们优势最大的领域——基准测试的核心主张由此在数据中得以印证。长时记忆能力同样强劲，对两个基线模型均达到77.8%的胜率。这一优势部分源于基线模型的结构性限制，而非单个薄弱回答：在视频通话模式下，豆包在约五分钟无语音输入后会自动挂断，Gemini则在约两分十五秒后挂断，因此在约半数记忆测试中，我们提出的问题落在这些截止时间之后，两个基线模型甚至不在场而无法回应，得分为零。然而，在那些处于其会话时长限制内的较短案例中，我们的模型依然获胜或打平。我们将此归功于长时记忆设计：通过中期与长期两个步骤，持续压缩、合并并组织模型所看到的内容，使得当问题最终到来时，流中更早的信息仍能触手可及。
基线模型取得优势的地方，其原因各不相同，且从来都不是在时机把握上。豆包的主要优势领域是实时解说与指导，在此类任务中，它赢得了22.2%的案例，另有22.2%打成平手，而我们则占55.6%。它的优势在于质量而非时机：更大的模型规模赋予了它更广博的知识、更丰富的风格和更多样的措辞，这使其在某些叙述任务中处于舒适区。事实上，时机把握是它的弱点。在解说任务中，它的时机把握反而成了劣势：回复在时间上很不稳定，有些段落过于频繁，有些段落又过于稀疏，因为其周期性的外部触发机制无法判断何时才真正需要发言。这印证了我们的核心设计原则：决定何时回应必须是模型的原生能力，由模型内部学习和判断，而非由外部触发机制提供的行为。因此，豆包只有在质量优势足够大，足以在等权重评分下弥补其较弱的时机把握时，才能在这些案例中胜出。而我们自己的失利则仅限于质量问题：模型在解说过程中偶尔会出现幻觉，我们将这一局限主要归因于其参数规模，并将其视为未来工作的主要方向。Gemini的唯一优势领域是时间感知，这也是与之竞争最激烈的维度，在此维度上，它打平了40%的案例，赢得了10%，将我们的胜率限制在50%，而在所有其他场景中，它未能赢得哪怕一次对比。原因在于该场景的特殊性：某些时间感知案例是由用户触发的，问题只在相关时刻过去之后才被提出，因此实时性要求较低。在这些事后提问中，Gemini强大的底层模型仅凭回答质量就能表现良好。在这两种情况下，基线模型都只在时间压力最低时才能缩小差距，依靠原始回答质量或针对那些不再需要即时响应的问题取胜，而恰恰在回复必须落在正确时机的地方，它们落后得最远。
案例研究。
为将量化差距具体化，我们逐一剖析了六个代表性案例；完整的并排视频对比可在我们的博客中查看。222https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction 每个案例都将 JoyAI-VL-Interaction 与豆包和 Gemini 的应用程序内视频通话助手在相同输入下进行了对比。
在跌倒检测警报案例（博客中的“02 监控与告警”部分）中，一个人倒在摄像头前。JoyAI-VL-Interaction 在跌倒发生的瞬间就发出警报，而豆包在四到五秒后才做出反应，Gemini 则明确承认自己完全无法监控现场。这种延迟并非偶然：它是外部触发器的轮询间隔，在不容许任何延迟的事件上表现为延迟。实时飞镖计数案例（博客中的“05 实时计数”部分）测试了持续的、与事件锁定的计数能力。JoyAI-VL-Interaction 在每支飞镖击中靶面时精确递增计数，准时记录了全部六次投掷，而豆包只计了两次，且存在明显延迟，Gemini 则只说了一句“让我检查一下”后就陷入沉默。街头采访翻译案例（博客中的“01 实时翻译”部分）探究了连续性，任务要求实时翻译采访画面中的字幕，而非音频。JoyAI-VL-Interaction 持续翻译语音，即使画面字幕中出现汉语拼音时也能保持准确，而豆包和 Gemini 都只翻译了请求发出时可见的内容，然后便停止，将一个持续进行的任务当作单次交互来处理。定时烹饪场景案例（博客中的“06 时间感知”部分）要求系统在二十秒间隔结束后发出信号，这是对内部时间感的直接测试。JoyAI-VL-Interaction 仅偏差一到两秒；豆包完全未能在时间点发出信号，而 Gemini 大约在四十秒时才发出信号，几乎是目标时间的两倍。宠物直播解说案例（博客中的“04 实时解说”部分）播放一段由不同宠物短片组成的连续流，要求模型跟上场景变化，识别并叙述每只动物的情绪状态。JoyAI-VL-Interaction 在每只宠物出现时进行解说，而豆包无法跟上，只描述了十三个中的三个，且描述不准确，Gemini 则仅在收到提示时评论了一次。手机应用委托案例（博客中的“09 智能体委托”部分）展示了一项基线模型完全不具备的能力：当手机的应用界面出现在屏幕上时，要求模型生成能渲染出类似界面的 HTML。JoyAI-VL-Interaction 识别出这超出了实时推理的范围，将任务交给后台模型，并返回能复现所观察界面的可用 HTML，全程无需离开实时会话。
涌现能力。
还有两个案例更为引人注目，因为它们展现出了模型从未被训练过的能力。在“购物应用引导”案例（博客中的03应用引导）中，用户在手机屏幕上滑动，追求一个购物目标。JoyAI-VL-Interaction 能够即时响应，并随着每次滑动同步进行解说，引导用户找到目标商品；而豆包和 Gemini 均未能主动或实时地做出反应，仅对最初显示的少数几个屏幕进行了描述。值得注意的是，我们的训练数据中不包含任何类型的应用界面视频：这种跟踪变化界面的能力是模型自主习得的，泛化到了一个它从未见过的领域。在“旅行场景解说”案例（博客中的04实时解说）中，系统被要求每四秒解说一次。JoyAI-VL-Interaction 全程保持这一节奏，且解说内容始终充实有力；豆包仅解说了开头场景约二十秒，完全忽略了四秒一次的指令；而 Gemini 尽管收到的是中文指令，却用英文回复，且仅评论了一次。该任务结合了定时动作和实时解说两种能力，而这两种能力在我们的训练数据中从未同时出现过：模型在推理时将它们组合起来，这是一种能力的涌现性交叉，而非它被展示过的模式。
在这些案例中，同一个因素将 JoyAI-VL-Interaction 与那些规模大数十倍甚至数百倍的系统区分开来：时机。通过将“何时回应”这一决策内化为模型自身习得的能力，而非外部触发器强加的行为，JoyAI-VL-Interaction 能在每个事件所需的时刻做出反应；而那些规模更大、但基于轮次或轮询的系统，要么反应迟缓，要么单轮交互后便停止，要么根本不参与。此外，其紧凑的规模并未对其最终能处理的任务设置上限：当某个问题确实需要更强大的模型时，JoyAI-VL-Interaction 不会尝试在线回答，而是将其委派出去——通过异步请求将困难子任务发送给后台模型，待结果返回后再整合进来，同时始终保持与用户的实时互动。因此，一个紧凑的 8B 模型得以将实时、精准的临场感与按需获取的深度推理能力相结合，我们认为这正是它能在流式场景中最重要的维度上超越远大于它的模型和成熟产品的决定性原因。
5.3 局限性
对我们比较结果的公正解读，必须从规模说起。我们评估所对比的两款视频通话助手，其背后分别是驱动豆包的 Seed 2.0 和驱动 Gemini 的 Gemini-3.1-flash-live，两者都是远大于我们的模型；而且它们都是经过长期针对真实用户和使用场景调优的成熟产品。相比之下，JoyAI-VL-Interaction 是一个紧凑的 8B 规模模型。因此，我们预期——并且也不声称相反——在通用的轮次交互能力上，这些产品比我们更强：它们拥有更广泛的世界知识、更精良的一对一聊天体验，以及在处理复杂或罕见输入时更强的鲁棒性。
我们的主张范围更窄，但我们认为也更具说服力。在本文所针对的六种事件驱动场景中——这些场景涉及实时在场、主动交互和时间感知，并且已经具备丰富的部署价值——一个规模小得多的模型仅仅因为是交互式模型而非轮询式模型，就天然占据了优势。一个紧凑的开源模型能够在这一领域胜过规模大得多且经过重度优化的产品，在我们看来，这正是关键所在：交互性本身作为一种能力值得规模化扩展，而且这种扩展在达到巨大规模之前就已经开始产生回报。
本次发布所依赖的数据与评估，均是刻意为之，仍处于早期阶段。在数据方面，我们尚未对数据混合比例进行调整，也未将其扩展至最终预期的规模，且还需进行新一轮的数据清洗；这一情况已体现在我们的结果中——相对稀疏的解说数据，加上模型本身规模较小，是导致实时解说过程中偶尔出现模型幻觉的主要原因（见§5）。评估同样属于初步性质，仅涉及六个场景和58个由人工评分的案例，对比了两款产品，而非我们最终期望的更大规模、更精细的研究。尽管如此，我们仍选择在此阶段发布，原因对数据和评估两方面而言是相同的：即便使用这些早期数据和有限的评估，该模型已展现出我们从未明确训练过的交互能力。亲眼目睹这些能力的涌现，比任何单一数字都更能让我们确信：交互性本身就是一个值得规模化发展的方向。正是基于这一信念，我们宁愿现在就将模型、配方和系统交到社区手中，而不是等到打磨出完美的数据配方和更大的基准测试后再发布：我们渴望与大家共同推进这一方向，而非闭门造车。后续版本将调整数据混合比例、扩展语料库，并报告更广泛、更系统的评估结果；在此期间，我们期望他人最能够直接借鉴和拓展的，是数据构建的方法论以及这一方法本身，而非特定的语料库或基准测试。这些局限性也指明了前行的道路：缩小通用能力和鲁棒性方面的差距，拓宽应用场景，并进一步探索交互模型的能力边界。
6 结论
我们致力于推动大型模型超越轮询式交互范式——即模型仅在收到指令时才“睁眼”响应——转向真正的流式交互，让模型持续存在于现实世界中，自主决定何时采取行动。为此，我们发布了据我们所知首个开源视觉驱动交互模型，并附带了构建该模型所需的一切资源：时间对齐数据、训练方案、8B参数模型，以及一套完整可部署的系统。
其核心信念只有一个：交互能力应作为模型自身的可扩展能力，而非通过外部框架模拟实现。扩展交互能力意味着同时扩展三个维度：模型在值得发声的时刻自主发言的自由度、实时交互的能力，以及对时间流逝的感知能力。随着这些能力的提升，模型将不再像等待查询的工具，而更像一个自然在场的参与者——观察正在发生的事件，并像人类一样做出回应。我们认为这是与智能扩展并行的独立扩展方向，将使模型更贴近真实的人类场景。
在我们研究的各类流式场景中，已经可以观察到早期迹象：交互模型在真实部署中天然具备优势，无论是监控与实时解说，还是逐步指导与陪伴。我们所追求的那个时刻，是日常生活中的一幕：当你结束漫长的一天，疲惫地回到家，还没来得及开口，一个轻柔的声音便察觉到你的状态，主动说道：“我看得出你累了，今天一定很辛苦吧。”这种不请自来的在场感，正是交互模型所能实现的，而传统的轮询模型——必须等待被召唤——永远无法做到。无论是模型本身还是系统层面，都还有许多开放问题，我们已公开了整个技术栈，以降低门槛、加速这一转变。轮询模型、产品及其优化已经积累了多年，而基于视觉的交互模型才刚刚起步；如果这次发布能为它们在那些早已稳固的高峰之间开辟出第一个立足点，并吸引更多人加入我们共同构建，那便完成了它的使命。我们邀请社区与我们一同探索，一个真正“在场”于世界中的模型，究竟能成为什么。
参考文献
- Bai et al. [2025] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. https://arxiv.org/abs/2511.21631.
- ByteDance Seed [2026] ByteDance Seed. Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity. Available at ByteDance Seed Model Cards, 2026.
- Chen 等人 [2024] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, 以及 Mike Zheng Shou。Videollm-online：面向流式视频的在线视频大语言模型。发表于 CVPR，2024 年。
- Chien 等人 [2026] Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, 以及 Alexandre Défossez。Moshirag：面向全双工语音语言模型的异步知识检索，2026 年。https://arxiv.org/abs/2604.12928。
- Du 等人 [2025] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Xian Shi, Keyu An 等人。Cosyvoice 3：通过规模化扩展与后训练实现野外语音生成。arXiv 预印本 arXiv:2505.17589，2025 年。
- Gao 等人 [2023] Zhifu Gao 等人。Funasr：一个基础的端到端语音识别工具包。发表于 INTERSPEECH，2023 年。
- Google [2026] Google。Gemini 3.1 Flash Live：让音频 AI 更自然、更可靠。Google 博客（The Keyword），2026 年 3 月。https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-live/。发布于 2026-03-26。访问于 2026-06-06。
- Guan 等人 [2026] Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, 以及 Xiang Bai。视频流式思考：VideoLLM 可以同时观看与思考。arXiv 预印本 arXiv:2603.12262，2026 年。
- Hou 等人 [2026] Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, 以及 Jiaqi Wang。Adacodec：面向视频多模态大语言模型的预测性视觉编码，2026 年。https://arxiv.org/abs/2606.02569。
- Kwon 等人 [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, 以及 Ion Stoica。基于 PagedAttention 的大语言模型服务高效内存管理，2023 年。https://arxiv.org/abs/2309.06180。
- Lab [2026] Thinking Machines Lab。交互模型：一种可扩展的人机协作方法。Thinking Machines Lab：联结主义，2026 年 5 月。10.64434/tml.20260511。https://thinkingmachines.ai/blog/interaction-models/。
- Li 等人 [2025] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, 和 Jiaqi Wang。Ovo-bench：你的视频大语言模型距离真实的在线视频理解还有多远？，2025。https://arxiv.org/abs/2501.05510。
- Lin 等人 [2024] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, 和 Maosong Sun。Streamingbench：评估多模态大语言模型实现流式视频理解的差距，2024。https://arxiv.org/abs/2411.03628。
- Liu 等人 [2026] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, 和 Pavlo Molchanov。GDPO：面向多奖励强化学习优化的分组奖励解耦归一化策略优化，2026。https://arxiv.org/abs/2601.05242。
- Lu 和 Lab [2025] Kevin Lu 和 Thinking Machines Lab。在线策略蒸馏。Thinking Machines Lab: Connectionism，2025。10.64434/tml.20251026。https://thinkingmachines.ai/blog/on-policy-distillation。
- Nous Research [2026] Nous Research。Hermes Agent：一个自我改进的开源 AI 智能体。https://github.com/NousResearch/hermes-agent，2026。GitHub 仓库。
- OpenAI [2026] OpenAI。通过 API 中的新模型（GPT-Realtime-2）推进语音智能。OpenAI 博客，2026 年 5 月。发布于 2026-05-07。https://openai.com/index/advancing-voice-intelligence/。访问于 2026-06-02。
- OpenClaw Foundation [2026] OpenClaw Foundation。OpenClaw：一个开源自主 AI 助手。https://github.com/openclaw/openclaw，2026。GitHub 仓库。
- Qian 等人 [2025a] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, 和 Jiaqi Wang。Dispider：通过解耦感知、决策与反应，赋予视频大语言模型主动实时交互能力。arXiv 预印本 arXiv:2501.03218，2025a。
- Qian 等人 [2025b] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, 和 Jiaqi Wang。利用大语言模型进行流式长视频理解。神经信息处理系统进展，37:119336–119360，2025b。
- Qin 等人 [2026] Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Dingyu Yao, Zheng Lin, Peng Fu, Nan Duan, 和 Jiaqi Wang。Easyvideor1：更简单的视频理解强化学习，2026。https://arxiv.org/abs/2604.16893。
- Qwen 团队 [2026] Qwen 团队。Qwen3.5-omni 技术报告，2026。https://arxiv.org/abs/2604.15804。
- Shao 等人 [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, 和 Daya Guo。Deepseekmath：在开放语言模型中突破数学推理的极限，2024。https://arxiv.org/abs/2402.03300。
- 火山引擎 (2026) [字节跳动] 火山引擎（字节跳动）。实时对话式 AI：视频与图像理解。火山引擎文档，2026。https://www.volcengine.com/docs/6348/1408245。访问日期 2026-06-02。
- Wang 等人 [2026] Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, 和 Linfeng Zhang。通过分层 token 压缩加速流式视频大语言模型，2026。https://arxiv.org/abs/2512.00891。
- Wang 等人 [2024] Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, 和 Dongyan Zhao。VideoLLM 知道何时说话：通过视频-文本双交互格式增强时间敏感型视频理解，2024。https://arxiv.org/abs/2411.17991。
- Xia 等人 [2025] Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, 和 Kaiyang Zhou。流式视频指令微调，2025。https://arxiv.org/abs/2512.21334。
- Xu 等人 [2025] Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, 和 Song Han。StreamingVLM：无限视频流的实时理解，2025。https://arxiv.org/abs/2510.09608。
- Yan 等人 [2026] Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Hao Liao, Xing Xie, Tao Jin, 和 Jianxun Lian。Proact-VL：用于实时 AI 伴侣的主动式 VideoLLM，2026。https://arxiv.org/abs/2603.03447。
- 杨等人 [2025] 杨安、李安峰、杨宝松、张北辰、惠彬源、郑波、于博文、高畅、黄成根、吕晨旭、郑楚杰、刘大恒、周凡、黄飞、胡峰、葛浩、魏浩然、林欢、唐嘉隆、杨健、涂建红、张建伟、杨建新、杨嘉熙、周静、周景仁、林俊阳、党凯、鲍克勤、杨可欣、余乐、邓良浩、李梅、薛明峰、李明泽、张沛、王鹏、朱琴、门睿、高瑞泽、刘世轩、罗双、李天浩、唐天一、尹文彪、任兴章、王新宇、张新宇、任选成、范阳、苏阳、张一昌、张英儿、万宇、刘玉琼、王泽坤、崔泽宇、张振如、周志鹏、邱子涵。Qwen3 技术报告，2025 年。https://arxiv.org/abs/2505.09388。
- 姚等人 [2026] 姚定宇、顾书焕、司庆一、周俊豪、杨晨旭、秦传宇、顾乃斌、林政、王伟平、段楠、王嘉琪。利用野外流式视频，2026 年。https://arxiv.org/abs/2606.08615。
- 姚等人 [2025] 姚林丽、李一程、魏远成、李磊、任树怀、刘元新、欧阳坤、王利安、李世成、李思达 等。Timechat-online：流式视频中 80% 的视觉 token 天然是冗余的。载于《第 33 届 ACM 国际多媒体会议论文集》，第 10807–10816 页，2025 年。
- ydyhello [2026] ydyhello。Awesome-VLM-Streaming-Video：流式视频视觉语言模型论文与开源代码精选集。GitHub 仓库，2026 年。https://github.com/ydyhello/Awesome-VLM-Streaming-Video。访问日期：2026-06-06。
- 曾等人 [2026] 曾翔宇、邱科帆、张庆宇、李新浩、王静、李佳欣、闫子昂、田坤、田蒙、赵新海、王毅、王利民。Streamforest：基于持久事件记忆的高效在线视频理解。载于《第三十九届神经信息处理系统年会》，2026 年。https://openreview.net/forum?id=9loSPaBwGO。
- 张等人 [2026] 张浩伟、杨树东、傅金兰、吴锡炯、邱锡鹏。Hermes：将 KV 缓存作为分层内存用于高效流式视频理解，2026 年。https://arxiv.org/abs/2601.14724。
- Zheng 等人 [2024] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, 以及 Ying Sheng. Sglang: 结构化语言模型程序的高效执行, 2024. https://arxiv.org/abs/2312.07104.
7 附录
7.1 训练数据示例
[*]Equal Contribution \contribution[†]Project Lead \contribution[🖂]Corresponding Authors ]JD.com \checkdata[Project page]https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction/ \checkdata[Repository]https://github.com/jd-opensource/JoyAI-VL-Interaction \checkdata[Release]The model weights, interaction data, and full system code will be released by June 20, 2026.
JoyAI-VL-Interaction:
Real-Time Vision-Language Interaction Intelligence
Dingyu Yao
Junhao Zhou
Chenxu Yang
Chuanyu Qin
Haowen Hou
Zheming Liang
Congcong Wang
Yuhang Cao
Shenglong Ye
Shuai Xie
Shuhuan Gu
Haoyang Huang
Qingyi Si
Nan Duan
Jiaqi Wang
Abstract
Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Once missed, the moment is gone, because the physical world does not pause. Yet today’s large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. This opens a "watch-and-do" mode of human-AI collaboration, expands AI assistance to far more real-life situations, and aligns naturally with the real-time demands of embodied intelligence. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model that is proactive by design. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video, from cameras, livestreams, or security feeds, into the model, making it genuinely present in the world. It supports hours of continuous video with sub-second latency. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API, model, or agent. Across six real-world streaming scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin on both quality and timing, winning 77.6% against Doubao and 87.9% against Gemini. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system. With our repository, anyone can deploy an assistant that sees the present, speaks up on its own, and moves fluidly between the physical and digital worlds. Our goal is to move VLMs beyond turn-based dialogue toward real streaming interaction, openly and together.
1 Introduction
The moments that most need AI are often the very moments no one has time to ask for it. A toddler wanders toward a hot stove. The decisive play in a match is over before anyone has time to cheer. An aging parent living alone falls in the next room. What these moments share is not merely urgency, but timing: the instant that calls for a word, a cheer, or a steadying hand arrives before anyone thinks to ask, and disappears just as quickly. Today’s large models are not built for such moments. Most remain fundamentally turn-based: they wait to be addressed, and respond only after they are asked. This limitation is structural, not merely a matter of speed. A turn-based model cannot perceive “now” by construction: it opens its eyes only at the moment it is prompted. What these situations require instead is a model that continuously sees what is happening in the present and interacts with the world as events unfold.
Existing efforts do not truly solve this problem. One line of work is real-time omni models, such as GPT-Realtime-2 [17] and Qwen3.5-Omni [22], which are genuinely end-to-end single models (audio and video in; speech out) with low latency and no stitched-together pipeline. However, their optimization target remains conversational turn-taking: responding as quickly as possible after the user has spoken. Fundamentally, they are still organized around dialogue. They wait for the user’s turn, and are not designed to continuously watch a visual stream and decide on their own when the moment calls for speaking. Another line of work is consumer video-call products. Doubao’s in-app video call [2] chases proactivity by firing a background request every few seconds, but its core is still turn-based and its autonomy is locked to the polling cycle: an event is not observed until the next poll, so the system can never react sooner than that interval. Gemini’s in-app video call [7] goes even less far: it does not perform the background polling that Doubao relies on to monitor for events, and instead remains strictly in a one-question, one-answer interaction pattern. A third line is research on streaming-video understanding [19, 20, 3, 26, 29, 8, 33], which mostly remains at the laboratory stage. It is often evaluated in the manner of offline video understanding [12, 13], and it rarely brings together the properties a real-world deployment demands [31], namely proactive response[27, 19, 29], long-horizon memory [32, 35, 34], and real-time operation [28, 25]. It typically advances only one dimension at a time. In short, none of these systems meets the needs of real deployment: each is either turn-driven rather than event-driven, not genuinely present in the live world, or alive only inside a benchmark.
JoyAI-VL-Interaction is built for exactly this. We adopt the term interaction model [11] from Thinking Machines Lab (TML) and make its criterion explicit: an interaction model judges for itself, moment to moment, when the time is right to respond to the ongoing stream, the way a person does, choosing when to speak and when to stay silent rather than answering only when addressed. A turn-based model, however low its latency or native its architecture, cannot choose its own moment: it speaks only when a user’s turn arrives. The difference is not how fast a model answers, but whether it decides for itself when answering is worth it. We and TML recognized at nearly the same time that interactivity is worth scaling as a capability of its own; where we part is the route to it. TML fuses speech and vision into one large model and shared a research preview; we make vision the first-class driver with speech as pluggable I/O, keep the model compact enough to run on local, low-cost hardware, and open the whole stack so anyone can deploy, reproduce, and build on it.
Contribution 1 — the VL-interaction model. Our central idea is simple: make when to act a learned, per-second decision of the model itself, with staying silent treated as a first-class action alongside speaking and delegating. We build this vision-first interaction model on our base vision-language model, JoyAI-VL 1.0 (§3.1). Speaking and staying silent are the root of proactivity and time awareness, knowing when a moment is worth a word and when it is not; delegating is how the model reaches for more capability when a problem outgrows real-time inference. We pair every second of the visual stream with the action it calls for, yielding time-aligned data, which we construct at scale (§3.2); from this the model learns its interactivity, as a behavior of its own rather than something bolted on by external turn-detectors or activity heuristics. To keep it affordable over an unbounded stream, we encode the video with AdaCodec[9], which spends far fewer tokens on each predictable frame so the budget grows far more slowly over a long stream (§3.1). We open-source the model weights, the training recipe, and the data behind it.
Contribution 2 — the VL-interaction system. We open-source a complete system that works out of the box: anyone can feed it a webcam or a livestream, and the interaction model immediately sees what is happening and interacts with the user in real time, genuinely present in the scene. Around the model, the system provides everything a deployment needs: ASR and TTS, a visualization UI, long-horizon memory, and a background bridge to any API, model, or agent the user brings, all served on standard vLLM to stay real-time over hours of continuous video. Among these, the interaction model alone decides whether and when to interact; the rest only transduce and orchestrate around it. Each ships with an open-source default, and the system provides the hooks to replace it, so a deployment can drop in its own ASR/TTS API or custom modules without rebuilding the stack. The system runs two concurrent loops, joined by the model’s “delegate” action: a real-time loop with the user, and an asynchronous loop in which the model offloads a hard problem to a background brain while staying present with the user, folding the result back in when it returns. In a head-to-head human evaluation across six everyday scenarios, JoyAI-VL-Interaction is preferred over both Doubao’s and Gemini’s in-app video-call assistants by a wide margin on quality and timing, winning 77.6% of comparisons against Doubao and 87.9% against Gemini. Its strongest margins fall on the most time-critical settings: it wins every comparison in monitoring and alerting against both systems and never loses one in real-time translation or counting, exactly the event-driven, act-at-the-right-moment tasks that turn-based products structurally miss (§5).
The value here is more than a better “video assistant.” When a model can be present and decide for itself what to watch and when to speak, human–AI collaboration shifts from “submit a request and wait” to “watch-and-do,” and AI can step into far more of everyday life, honoring embodied intelligence’s premise that the physical world cannot be paused. The same capability is the substrate for a class of long-awaited products: a companion that is truly present, AI glasses that translate or caption what you see, an aide that serves as eyes for those who cannot, and uses no one has imagined yet. We release JoyAI-VL-Interaction, the model, its training recipe and data, and the full system in the open, precisely so the community can build and explore all of this, and more, together. Our larger hope is to move the field from turn-based dialogue toward genuine streaming interaction, in the open.
2 Related Work
2.1 Turn-based Models and Products
Almost all of today’s models are turn-based: they act only when a user takes a turn, answering once addressed and otherwise idle. This is a structural property, not a matter of speed. A turn-based system has no mechanism for reacting to an event in the world that arrives with no accompanying user utterance, however quickly it can reply once spoken to. Two families of recent systems push hard on responsiveness while keeping this turn-based core.
Realtime and omni models.
OpenAI’s GPT-Realtime-2 reasons inside a single speech-to-speech loop [17], and Alibaba’s Qwen3.5-Omni [22] is a natively pretrained omni-modal model (text, audio, image, video) with real-time streaming and built-in turn-taking and interruption handling, the latter openly released. These remove the latency of cascaded stacks, but what they optimize is conversational turn-taking: how quickly and naturally they answer once the user has spoken. Their interaction is organized around dialogue and waits for the user’s turn; they are not built to watch a visual stream and decide, unprompted, that this is the moment to speak. Being open, real-time, and multimodal (as Qwen3.5-Omni already is) is therefore necessary but not sufficient for the setting we target.
Consumer video-call products.
In-app “video call” features are the most familiar real-time multimodal assistants and our most recognizable baseline, yet their cores remain turn-based. Doubao’s runs on ByteDance’s Volcano Engine conversational-AI stack, a configurable ASR/VLM/TTS pipeline with stop/turn detection, and its documentation makes the mechanism plain: while no one is speaking, the server only caches incoming video frames and does not send them to the model, so the application must periodically fire an ExternalTextToLLM trigger to obtain any analysis [24]; this characterization is based on the publicly available version of the stack, which may differ from the deployed product. “Monitoring” is thus an external clock bolted onto a turn-based model: reaction to an on-screen event waits for the next trigger and can never beat the polling interval. Gemini’s video call goes less far still; in our use it answers only about the frame visible the instant a question is asked. In neither case does the decision of when to respond live in the model.
2.2 Interaction Models
A distinct, newer line breaks with the turn-based assumption by moving interactivity into the model.
TML interaction models.
Thinking Machines Lab recently named this an interaction model and argued, as we independently and concurrently concluded, that scalable interactivity should come from the model itself rather than an external harness of turn detectors and activity heuristics. Their model handles audio, video, and text in real time and can speak unprompted, acting on the ongoing stream rather than waiting for a user’s turn, and, as in our system, it is paired with an asynchronous background model for slower, harder reasoning, the two sharing context; it was released as a research preview [11]. That two groups arrived at this approach at nearly the same moment we read as a sign that the move from turn-based to interactive models is a direction whose time has come. Within this shared interaction–background design, our aesthetic differs in two deliberate ways. On modality, rather than fusing audio and vision into the model alike, we keep speech (ASR/TTS) a pluggable module and make vision the model’s intrinsic, primary modality for proactive interaction, fitting a watch-and-interact setting in which voice is interchangeable I/O. On scale, TML-Interaction-Small is a 276B-parameter mixture-of-experts model (12B active) that its authors describe as the smallest they can serve at the required latency, whereas we deliberately choose a compact 8B model (balancing interaction-grade responsiveness, local and low-cost deployability, and capability) so that it can be run, fine-tuned, and reproduced widely rather than only on large infrastructure.
Full-duplex speech.
For speech, Kyutai’s MoshiRAG [4] is the prominent open instance: a real-time, full-duplex speech–text model in which both sides can speak and listen at once rather than strictly alternating, and the closest open precedent for putting interaction in the model. Its modality and scope (spoken conversation) set it apart from the visual, event-driven setting we address.
2.3 Streaming Video Understanding
A research line lets models process video as it arrives rather than after the fact, spanning streaming video LLMs, proactive response [27, 19, 29], real-time inference [28, 25], and long-horizon video memory [32, 35, 34]. It is closest to our model contribution, but three gaps separate it from a deployable interaction model. It typically advances one property at a time (responsiveness, or proactivity, or memory) rather than the three together. It is usually measured on offline benchmarks, at times even in the manner of offline video understanding, which never tests reaction to live events under real-time constraints. And it stops at the model, without the surrounding system (serving, memory, transduction, delegation) that hours of sustained real-time presence demand.
2.4 Our Position.
Read along two axes, how a model interacts and what is released, JoyAI-VL-Interaction occupies a cell no prior work does. Against turn-based systems, native or product, the decision of when to act lives inside our model and is taken every second, event-driven, so reaction is bounded by inference rather than by a user’s turn or a trigger clock. Against TML’s interaction model, we make vision the first-class driver, the watch-and-interact setting rather than audio–video conversation, and we open-source not a research preview but the model, its training recipe, and a complete, composable, deployable system. Against MoshiRAG, we are vision-driven rather than speech-centric, reacting to events that carry no conversational turn at all. And against streaming-video research, we bring responsiveness, proactivity, and memory together in one model and run it inside a system built for sustained presence. To our knowledge, JoyAI-VL-Interaction is the first open, vision-driven interaction model released together with a complete deployable system: the point where all of these properties meet, rather than any one alone.
3 JoyAI-VL-Interaction Model
Figure 1 summarizes the model. JoyAI-VL-Interaction watches a streaming visual input and, at every second, takes one of three actions: speak to the user, stay silent and keep watching, or delegate a slower, harder task to an asynchronous background model whose result is folded back into the stream when ready. The first two actions form a real-time loop with the user; the third, an asynchronous loop with the background.
Two design points frame the rest of this section. First, the model learns a background-agnostic delegation protocol, so the background can be swapped for any external model, agent, or API; we cover the model side here and the system side in §4. Second, the model does not process or generate speech itself: turning voice into text and back is left to pluggable ASR/TTS in the system, keeping the autonomous, vision-driven core decoupled from interchangeable I/O.
The rest of the section builds the model from this behavior outward: the base model and architecture (§3.1), the data that teaches when to speak, stay silent, and delegate (§3.2), and the training recipe (§3.3), which covers continue training, the training objective, reinforcement learning, and infrastructure.
3.1 JoyAI-VL 1.0 and Architecture
JoyAI-VL 1.0.
JoyAI-VL-Interaction is built on JoyAI-VL-1.0, where the language model is initialized from Qwen3-8B [30], the visual encoder is the Qwen3-VL ViT [1], and the projection layer between them is trained from scratch. JoyAI-VL 1.0 is obtained through three stages, representation alignment, vision-language pre-training and post-train with On-Policy Distillation [15] and RL [14, 23]. At this point the model is a conventional, turn-based VLM. Its per-second interaction behavior, deciding each second whether to speak, stay silent, or delegate, is acquired afterward through the interaction-training recipe in §3.3.
Native Streaming Video Codec.
We tokenize the video stream with AdaCodec [9] (Figure 2), a predictive visual code inspired by predictive coding and video codecs: instead of re-encoding every frame, it transmits only what prediction cannot explain. It spends full ViT tokens only on reference frames and encodes the predictable frames in between as compact P-tokens built from motion and residuals, so the model reads an interleaved stream of reference tokens and P-tokens rather than a sequence of full RGB frames. A predictive-cost reset opens a new reference frame whenever prediction becomes costly, placing reference frames exactly where the scene changes enough to need them. This matches our setting directly: a present, always-watching model must consume an unbounded video stream, where a per-frame interface spends full ViT tokens on every frame, so cost and latency grow quickly with the length of the stream. AdaCodec spends only about 16 tokens on each predictable frame and full ViT tokens only at scene changes, so the budget grows far more slowly and its heavy part scales with how much the scene changes rather than with frame count.
3.2 Data Construction for VL-Interaction
The applications we build for share a single demand: the assistant must be present and act on its own, at the moment that matters, with no one there to prompt it. A security feed where a flame has just appeared, a livestream where the item a viewer wants is flashing past, AI glasses that should caption or translate what is in front of you the instant it appears, a companion that should speak up only when there is something worth saying. In each case the right moment to act is set by the world, not by the user, and a model that waits to be addressed has already missed it. The data in this section is built to give the model exactly the abilities these settings demand: to watch a visual stream continuously and decide for itself, second by second, whether to speak, stay silent, or hand a hard problem to a heavier background model; to act on what it sees rather than wait to be asked; and to keep track of time, so that it knows not just what is happening but when. Teaching these abilities broadly, rather than for any single use case, is the goal of everything that follows.
Concretely, at each one-second step the model takes one of three actions over the visual stream seen so far: it can stay silent and keep watching, emitting a </silence> token; speak, emitting a </response> token and a textual reply; or delegate a slower, harder subtask to an asynchronous background model. Delegation follows a fixed, background-agnostic protocol defined over text requests and results, so any external model, agent, or API can serve as the background (§4). We teach these three decisions at one-second granularity, over and over, across as much of real life as we can capture.
Capabilities the data covers.
To span this range, the data comprises more than 4M time-aligned streaming clips organized into six families. The first five are: (1) proactive alerting and anomaly detection on live feeds; (2) time-aligned question answering in backward, present, and forward timing; (3) counting and perception over time; (4) live commentary and narration; and (5) multi-turn casual chat in a variety of styles, ranging from everyday conversation over egocentric short video, to extended dialogue across long videos, to companionship-style exchanges. The sixth (6) cuts across all of the others: delegation episodes pair the ongoing scene with hard, off-stream subtasks—video-grounded knowledge questions, STEM problems, and video reasoning problems—that the model should route to the background rather than answer inline. A single per-second action format applies across every family, so each scenario jointly teaches the model when to speak, when to stay silent, and, where appropriate, when to delegate.
Constructing time-aligned data.
Per-second supervision is what makes this behavior learnable, and it is costly to obtain at scale. Every example must be right along two axes: content, what the model says when it speaks, and timing, the exact second at which it should speak, stay silent, or delegate. Silence is a first-class label rather than the absence of one, since most steps in any stream should be silent and the model must learn to wait instead of speaking at every step. To get both axes right at scale, we run a multi-stage pipeline with dedicated verifier agents, and we deliberately favor quality over quantity. Each candidate example is checked at two levels: globally, over all input frames together with the full annotation, and locally, over the frames at the annotated timestamp and the reply tied to them. An example is admitted only if it passes both checks, and is discarded the moment it fails either. Each source is annotated along whichever axis is its bottleneck, and only examples that survive every check enter the corpus, so heterogeneous raw material is funneled into one clean, consistent per-second action format.
We tailor the construction to each family. For offline video-QA, the content of open-source data is already trustworthy, so the work is almost entirely about timing: a larger VLM marks when the evidence relevant to each question appears, and we form three timing types around it. In a backward example the question is asked after the evidence has appeared and the model responds at the moment the question arrives; in a present example the question is asked just as the evidence appears and the model responds at that moment; in a forward example the question is asked before the evidence appears and the model stays silent until the evidence arrives, then responds at that instant. For multi-turn casual chat, the casual nature of the exchange keeps the bar on content low, so we prioritize timing instead. Two VLM agents converse over a continuously playing video, one asking questions grounded in what it currently sees while the other watches the same stream and answers. To place each turn in time, we sample a time point at random and give the annotating model only the three frames around that point, which anchors every response to its timestamp and prevents large timing drift; a verifier agent then screens each exchange for grounding and quality.
For commentary and narration, the commentator’s authentic style is itself the value, so rather than synthesize it we collect open-source commentary and broadcast footage and recover the real spoken content with ASR, which preserves the natural cadence of when a person speaks and pauses and yields realistic silence labels. For counting, we run two passes over a counting-oriented source: the first keeps only videos in which objects reappear, and the second lays down the per-second labels. For perception over time, we insert time-conditioned content into ready-made multi-turn dialogues: for instance, requiring in the question that the answer be withheld for a precise interval before it is given, adding an instruction that increments a running count by one every seconds, or bounding the time window within which a question and its answer may occur; these rules yield exact, checkable timing without further manual annotation.
For alerting and anomaly detection, timing accuracy matters most, since an alert one second late describes a different moment, so we build this data from two sources. From open-source temporal-grounding annotations we convert directly into per-second labels, with a verification pass over the onsets. For web-collected videos, which carry no annotations, the pipeline proposes a candidate trigger window and tightens it through several stages: it filters videos to viable candidates, samples a window, generates and hard-filters a target by type, and applies a window-level verifier; a dense precheck then scans every 1 fps frame from the query time to the candidate trigger and discards the example if the target appeared earlier, so the alarm marks a genuine first onset, after which we localize the trigger to a single frame.
Anatomy of a delegation episode.
Delegation is the most intricate behavior in the corpus, and the most distinctive: it is where the model learns to recognize the limits of real-time inference and hand off, mid-stream, to a heavier background brain without ever pausing the world it is watching. Each episode is built end to end, pairing a trigger (a subtask the model should not solve inline) with a written request and a returned result, and we synthesize these episodes three ways: (i) inserting ready-made pure-text hard problems such as STEM questions, where it is the difficulty rather than the modality that should send them to the background; (ii) converting open-source offline video-reasoning problems and using a large model to weave them into annotated multi-turn chat, so a genuinely hard visual question can surface naturally in the middle of a conversation; and (iii) synthesizing full episodes with a multi-role agent pipeline, in which a Planner scripts the episode, a Timestamp / Visual Verifier runs multi-level checks, over the whole clip and at the trigger frame, that the trigger and its timing are grounded in what is actually on screen, a Background agent produces the heavy answer, and a Foreground Rewriter turns it into a natural reply.
At the heart of the episode is the two-loop choreography it teaches. The instant the model decides to delegate, it does not go quiet. It first hands the user a brief holding reply (for example, “let me look into that”), then emits a hidden delegate token and query that the user never sees, dispatching the hard subtask to the background while the visual stream keeps rolling. We then inject a random delay that simulates the background’s variable reasoning time, and only after that delay is the result folded back in, at which point the model produces its formal answer in context. The delay is the whole point: by varying it, we force the model to stay present while a delegation is still pending, continuing to watch the scene, field new turns, and hold its silence when nothing is worth saying, rather than freezing until the background returns. This is the exact seam where the real-time loop with the user and the asynchronous loop with the background are stitched together, and the model learns to run both at once. Across all six families, however different their raw material, every construction is reduced to the same per-second labels of silence, response, and delegation and passes through the same multi-level verification, which is what lets such heterogeneous sources train a single, unified interaction policy.
A transferable recipe, and signs of emergence.
Because the construction is defined by this common per-second format rather than by any one domain, the recipe transfers cleanly to new scenarios: adding a capability is as simple as supplying streams from that domain, annotated in the same per-second form and passed through the same verifiers, with no change to architecture or training objective. And the payoff scales. Even a modest amount of time-aligned data is enough for strong interaction behavior to emerge, a sign that the recipe is highly data-efficient and that time-aligned training is nowhere near saturated, with substantial headroom still ahead as the data grows. Most strikingly, capabilities we never explicitly trained for emerge on their own, including guiding a user through a purchase across changing app screens and improvising a full lecture from a slide deck. We read this as strong evidence that the model is acquiring a general watch-and-interact competence rather than memorizing scenario-specific tricks, and that the same recipe can grow the next generation of always-present assistants, from AI glasses that narrate what you see to companions and accessibility aides that serve as eyes for those who cannot. What we have seen so far is only the beginning.
3.3 Training Recipe of VL-Interaction
Continue training.
Starting from JoyAI-VL 1.0, a single supervised training stage turns the base into an interaction model; reinforcement learning then follows below. We mix the time-aligned interaction data of §3.2 into a large pool of conventional turn-based data and fine-tune on the combined corpus. This echoes §3.2: eliciting interaction is data-efficient, and time-aligned training is far from saturated.
Training objective.
On time-aligned data, silence steps vastly outnumber the steps on which the model speaks, so the supervised targets are dominated by the </silence> token. Under a standard SFT loss this imbalance pushes the gradient toward continued silence and dilutes the signal for responding. Following our streaming-native training [31], we therefore weight the assistant tokens by their role. Let be the supervised assistant-token positions and the control-token positions, where each control token is either </silence> or </response>. We assign to the first </silence> in a run, to a continued </silence>, and to </response>; every other position gets weight . We set , to down-weight silence continuations, and to up-weight response onsets. A delegation needs no separate weight, because it always rides inside a response. The objective is the normalized weighted cross-entropy
| (1) |
In practice we use and . We apply this weighted loss only to the time-aligned data; the conventional turn-based data is trained with the standard SFT loss.
Reinforcement learning.
Continue training teaches the three actions, but the finer points of timing are hard to perfect with token-level supervision: speaking at the right moment, holding silence when nothing is worth saying, and deciding when a subtask is better delegated than answered inline. We add a reinforcement-learning stage, run with GRPO, that optimizes the per-second policy directly against stream-level rewards. Because a long stream would otherwise unfold into hundreds of mostly silent turns, we keep rollouts tractable with answer-centered window sampling: for each gold response we build one trajectory that preserves streaming causality but retains only the turns that matter for timing, compressing the horizon from hundreds of turns to a handful and concentrating credit where the timing decision lives. The reward credits responses that are both correct and emitted within the right window, rewards appropriate silence, and rewards well-judged delegation; it penalizes false alarms (speaking or delegating with no cause), mistimed responses, and degenerate always-respond behavior. Delegation is scored in two parts: whether the model hands off the genuinely hard subtasks rather than easy ones it should answer inline, and whether it uses the returned result well once the background replies, including staying responsive while a delegation is still pending. Response content is additionally scored for quality by an LLM judge against task-specific rubrics, giving a consistent and fine-grained reward where simple rules fall short.
Infrastructure.
We run our RL stage on EasyVideoR1 [21], an efficient reinforcement-learning framework for training vision-language models on video. Its reward system is task-aware, with unified routing and modular extension, so rewards for text, image, video, and our streaming interaction tasks are served from a single pipeline.
4 JoyAI-VL-Interaction System
Around the interaction model of §3 we build a complete, deployable system whose single principle is this: the model is the only component that decides when to speak and when to delegate, while everything else (ASR, TTS, memory, the background brain, the visualization UI) is transduction and orchestration placed around it. Every component ships with an out-of-the-box open-source default and can be swapped for whatever fits a deployment, including the model itself, which any VLM trained with our recipe (§3.3) can replace. This "decision in the model, the rest replaceable" split is the system’s design thesis, and it buys two properties at once: the system is composable, so anyone can rebuild it for their own domain, and it is deployable today, because the periphery reuses standard infrastructure rather than a bespoke stack.
Compared with TML, which fuses speech into the model alongside vision, we keep the visual trigger, the model’s own decision to act on what it sees, as the native, in-model capability, and treat speech as interchangeable I/O. Placing interchangeable I/O outside the model is therefore not a weakness but a deliberate decoupling of the autonomous core from the parts a deployment will want to choose for itself.
The rest of this section builds it up: the two concurrent loops it runs, the background bridge that joins them, and how that bridge closes a loop from seeing to acting (§4.1); the pluggable transduction, visualization, and user-supplied modules around the model (§4.2); the long-horizon memory that keeps it coherent over hours (§4.3); and the vLLM-native serving path and open release that sustain sub-second presence and put the system in anyone’s hands (§4.4).
4.1 Two Concurrent Loops: from Seeing to Acting
The interaction model runs two conversations at once: a real-time loop with the user and an asynchronous loop with the background brain, joined by the model’s delegate action.
Ingestion.
The real-time loop begins at ingestion. A browser client pushes its camera stream to the server through WebRTC SDP negotiation, or submits an RTSP address that the server pulls; the two media paths are received and decoded server-side by aiortc and PyAV respectively, back into raw frames. A sampling module downsamples the stream at a fixed interval, 1 Hz by default and configurable per scenario, to balance temporal detail against real-time latency. The sampled frames are converted to RGB, JPEG-encoded, and wrapped as Base64 data-URLs, then passed with the active user query into an adaptation layer that assembles an OpenAI-compatible multimodal Chat Completions request, the same request form our vLLM serving path expects (§4.4).
The real-time and asynchronous loops.
These frames are fed to the model every second, so it always acts on what is happening now rather than on a delayed batch. Together with optional microphone audio transduced by streaming ASR into an active query, they drive the model: at every second it takes one action, to speak, stay silent, or delegate; when it speaks, streaming TTS renders the reply and the visualization UI reflects its state. When it delegates, control passes to the asynchronous loop, in which a background model runs against the background brain to handle slower, more demanding tasks.
The background bridge.
The edge that delegate points to is a background bridge, and the brain behind it defaults to the user’s own large-model API while remaining free to be any agent (such as Hermes Agent [16] and OpenClaw [18] ) the user brings. The scaffold supports this natively and defines a fixed, background-agnostic protocol over text requests and results (§3.2). The bridge exposes a background-agnostic text contract: the foreground emits a tagged query, which the bridge normalizes into a task-id, delegated question, foreground note, and bounded frame snapshot. It runs an isolated model or agent asynchronously under a fixed timeout; expiry is reported as an error event and in-flight work is cancelled on session cleanup. Results return as started/ready/error events, with full artifacts kept off-context and only a bounded digest woven back into the interaction model. Because the background can be an agent, the bridge also closes a loop from seeing to acting: the interaction model observes the physical world and, when warranted, delegates a task the background agent carries out in the digital world. This is the watch-and-do premise of §1 made operational: see, decide, act, in one system.
4.2 Pluggable Components: Transduction, Visualization, and Your Own Modules
ASR and TTS.
For speech, we ship a ready-to-run server built on open source ASR [6] and TTS models [5], and users can swap in their own model or API to match their language and preferred voice. These modules only convert modalities; they take no part in the interaction decision. Because speech has duration while generation does not, we predefine a simple rule to keep the two in step: while the previous utterance is still being spoken, the model’s next sentence is surfaced as text only and is not synthesized, so audio never piles up behind generation. For high-frequency settings such as live commentary, step-by-step guidance, and real-time subtitle translation, where output is near-continuous, we recommend running with TTS off, and we leave a fuller treatment of this regime to future work.
Visualization UI.
A built-in web interface, adapted from NVIDIA’s open-source live-vlm-webui111https://github.com/nvidia-ai-iot/live-vlm-webui, makes the running system visible and controllable.A left-hand configuration panel selects the video source—either a webcam or an RTSP livestream—and sets the sampling cadence, namely the number of frames per inference and the interval between inferences; it also offers a response mode that shifts the model’s style from a default balance to more talkative or more aloof. The center pane shows the live stream the model is watching. The right-hand pane is the conversation: it holds the running exchange with the model, including earlier turns, reports per-response and average latency in real time, and accepts either typed messages or spoken input through the microphone (ASR); when the model speaks, its reply is played back automatically through TTS. The interface doubles as a debugging and trust surface for sensitive deployments, and it can be replaced with a custom front end.
Bring your own modules.
Beyond these defaults, the system is open to user-supplied modules that enrich what the assistant perceives or remembers. As one example, a face-recognition module can let the assistant remember who you are and recognize when a particular friend or family member walks into the scene, opening up more personal and engaging interactions. There is a larger reason we open all of this. Turn-based interaction is a long-settled range, refined over years, with a vast ecosystem of systems and optimizations built up around it; VL interaction models, by contrast, have barely begun. By releasing the full stack, we hope to open a first breach for interaction models among those established peaks, and we invite the community to explore with us what this still-young way of working with AI can become.
4.3 Long-Horizon Memory
Long-horizon presence needs a memory whose footprint does not grow without bound as a stream runs for hours, and whose contents form a shared context read by both the streaming model and the background brain. We meet this with a pluggable hierarchical memory built on our prior work [31], organized into tiers at increasing compression. We organize the context into a three-tier hierarchy that partitions the stream into: (i) a short-term memory retaining the most recent seconds as raw vision tokens; (ii) a mid-term memory holding up to textual summaries of past short-term chunks, covering seconds at moderate compression; and (iii) a long-term memory storing up to aggressively compressed blocks, each consolidated from consecutive mid-term summaries and thus spanning seconds. Alongside these visual tiers, a dialogue memory keeps past queries and answers coherent across time. The consolidation steps run asynchronously, ahead of each boundary, so they hide behind mainline inference and never stall the real-time loop, and together the tiers reach up to roughly two hours of context. Because each tier above the buffer is stored as text, the memory also forms a stable per-chunk prefix that the serving path of §4.4 can cache and reuse; it is designed for exactly this, and it remains fully pluggable, swappable for any external memory a deployment prefers.
4.4 Serving and Runtime
vLLM-native serving.
Hours of continuous video defeat the obvious serving strategies. Recomputing attention over the full history saturates the context window within a few hundred seconds; a sliding window bounds the context but breaks prefix reuse, since successive steps no longer share a common prefix, so an engine’s prefix cache buys nothing and latency plateaus above the real-time budget; token pruning has the same problem. Most streaming-video methods inherit one of these and, built on plain Transformers, stay incompatible with efficient engines such as vLLM [10] and SGLang [36], which is much of why they remain in the lab. We instead designed the memory system around prefix reuse so that vLLM serves it directly: the text memory of §4.3 is prefilled once per chunk into a KV cache, and at every subsequent step only the newly observed frames and the previous reply are computed, while the memory and earlier in-chunk turns are reused without recomputation. AdaCodec (§3.1) further shrinks each predictable frame to about sixteen tokens, keeping per-step work small. Building the memory this way is precisely what lowers the bar to deployment: the system sustains over two hours of continuous video at sub-second end-to-end latency on standard vLLM.
Stateful sessions.
The runtime is organized around server-side sessions. Control messages, configuration updates, session resets, and state synchronization, travel as JSON over WebSocket or HTTP and carry a session_id, while the server keeps each session’s video context, question-and-answer trajectory, and memory state. When a request is assembled, this maintained context is injected automatically into the Chat Completions request, so a single round of client-server communication carries only the instantaneous observation, and all long-horizon context management lives at the backend. This is also what makes the prefix reuse above possible: the injected context is precisely the stable per-chunk prefix the engine caches and reuses.
Real-time robustness.
To hold latency steady under load, the runtime places timeouts, reconnection, and cancellation across connection setup, the WebSocket channel, RTSP signaling, and the auxiliary audio and video channels, and a session-level lock keeps requests from piling up; when it must, it drops stale frames or backfills late results into the correct interaction history by inference sequence number. The adaptation layer routes each request by its model identifier to the corresponding VLM service, runs the mid-term and long-term memory consolidation of §4.3 in parallel, and returns a standard JSON response.
Everything is open.
The model recipe, the background bridge, ASR/TTS, memory, orchestration, the visualization UI, and the vLLM serving path can all be replaced or extended in the repository. Releasing the model recipe and the complete system together is the whole point: from a single repository, anyone can stand up their own present, self-directed real-world assistant.
5 Experiments
5.1 Benchmark
We evaluate JoyAI-VL-Interaction not on offline video-understanding benchmarks but against the real, deployed products people would actually reach for, in the live, event-driven setting this paper is about. Concretely, we compare it head-to-head with the in-app video-call assistants of Doubao and Gemini, today’s most mature and recognizable real-time multimodal products (§2.1). We choose six everyday scenarios that fall squarely in the advantage zone of the interaction-model paradigm: settings that demand being present, acting unprompted at the right moment, and tracking events over time, precisely what a turn-based product, waiting to be addressed, is structurally unequipped to deliver however fast it answers once asked.
Scenarios.
The six scenarios are: (1) monitoring and alerting, flagging an event the instant it occurs; (2) real-time counting of objects or events over time; (3) real-time translation of on-screen content; (4) time awareness, acting on its own sense of elapsed time, for instance speaking once every few seconds on request or timing how long an activity lasts; (5) live commentary and guidance, narrating or walking the user through an unfolding scene at the right moments; and (6) long-horizon memory, answering about something seen far earlier in the stream. Underneath the six lie the three capabilities that define an interaction model: real-time operation, proactive response driven by what it sees, and long-horizon memory. The six are thus not an arbitrary set but a sampling of these axes, and the same capabilities reach far beyond this benchmark, into AI glasses, assistance for the blind, security monitoring, home robots, home and elder care, vision-grounded chat and companionship apps, live sports commentary, and delegation to background agents, among others. All are settings where being present and acting at the right moment is the whole task, the dimension on which turn-based products structurally fall short, so the six probe the paradigm gap itself rather than any single product feature. The six scenarios comprise 58 cases in total: 10 each for monitoring, counting, translation, and time awareness, and 9 each for commentary and memory. The cases are drawn mostly from public footage on the web, with a few recorded by us.
Baselines.
Our baselines are the video-call features of the Doubao and Gemini apps, the products we single out in §2.1 as turn-based at the core, each evaluated as it is actually deployed to end users. We use the app versions current in late May and early June 2026, and drive each product with the same input under matched conditions. We run JoyAI-VL-Interaction through our own system (§4) and compare it against each baseline separately, pairwise. The JoyAI-VL-Interaction system uses a three-tier memory with s, , and (§4.3). At evaluation time, our JoyAI-VL-Interaction system simulates offline videos as a live streaming source over RTSP, served by MediaMTX. Since Doubao and Gemini expose no system-level API, we drive them through their apps: we play the same video to each and pose identical questions at matched timestamps.
Protocol and metric.
For every case, human raters score each system on two equally important axes: quality, whether the response is correct, relevant, and well-formed; and timing, whether it arrives at the right moment, neither premature nor late, and whether the system stays silent when nothing is worth saying. Each axis is rated on a three-level scale, good, fair, or poor, and a system’s score for the case is the equal-weight average of its two axis ratings. The timing axis is the crux of the event-driven setting and the one turn-based products structurally struggle with (§1). To reduce bias, system identities are hidden from raters and the presentation order is randomized. Five raters carry out the ratings, all educated to at least the university level and working as researchers in LLMs, and inter-rater agreement is high. For each case we then compare the two systems’ weighted-average scores, which gives a win, tie, or loss for JoyAI-VL-Interaction. We report the per-scenario and overall win rate, the fraction of comparisons it wins, against Doubao and against Gemini respectively, alongside the tie and loss rates; a tie counts as neither a win nor a loss.
| Scenario | JoyAI-VL-Interaction | Tie | Doubao |
|---|---|---|---|
| Monitoring and alerting | 100.0% | 0.0% | 0.0% |
| Real-time counting | 70.0% | 30.0% | 0.0% |
| Real-time translation | 80.0% | 20.0% | 0.0% |
| Time awareness | 80.0% | 10.0% | 10.0% |
| Live commentary and guidance | 55.6% | 22.2% | 22.2% |
| Long-horizon memory | 77.8% | 22.2% | 0.0% |
| Overall | 77.6% | 17.2% | 5.2% |
| Scenario | JoyAI-VL-Interaction | Tie | Gemini |
|---|---|---|---|
| Monitoring and alerting | 100.0% | 0.0% | 0.0% |
| Real-time counting | 100.0% | 0.0% | 0.0% |
| Real-time translation | 100.0% | 0.0% | 0.0% |
| Time awareness | 50.0% | 40.0% | 10.0% |
| Live commentary and guidance | 100.0% | 0.0% | 0.0% |
| Long-horizon memory | 77.8% | 22.2% | 0.0% |
| Overall | 87.9% | 10.3% | 1.7% |
5.2 Results and Case Study
Experimental Results.
Across both comparisons JoyAI-VL-Interaction is the preferred system by a wide margin, and in every scenario it wins more comparisons than it loses. Against Doubao it is preferred in 77.6% of cases, ties 17.2%, and loses only 5.2%; against Gemini the margin is wider still, at 87.9% wins, 10.3% ties, and just 1.7% losses. The two baselines, in other words, almost never come out ahead overall.
Our model’s clearest advantage falls exactly on the vision-driven, time-critical scenarios that define the paradigm. In monitoring and alerting, the purest test of catching an event the instant it occurs, it wins every single comparison against both baselines (100%). Real-time translation and fast counting are similarly lopsided: it sweeps both at 100% against Gemini, and wins 80% and 70% against Doubao with no losses at all. These are precisely the settings where being present and acting at the right moment is the whole task, and they are where our margins are largest, which is the central claim of the benchmark borne out in the numbers. Long-horizon memory is also strong, at 77.8% against each. Part of this margin reflects a structural limit of the baselines rather than a single weak answer: in video-call mode Doubao automatically hangs up after about five minutes with no voice input, and Gemini at around two minutes fifteen seconds, so in around half of the memory cases the question we ask falls past these cutoffs and neither baseline is even present to respond, scoring nothing. However, on the shorter cases that stay within their session limits, our model still wins or ties. We credit it to our long-horizon memory design: a mid-term and a long-term step that continuously compress, merge, and organize what the model has seen, so that information from far earlier in the stream is still at hand when a question finally arrives.
Where the baselines do gain ground, they do so for different reasons, and never on timing. Doubao’s main foothold is live commentary and guidance, where it wins 22.2% of cases and ties another 22.2% against our 55.6%. Its edge here is one of quality rather than timing: a larger model scale gives it broader knowledge, richer style, and more varied phrasing, which lands it in the comfort zone of some narration tasks. Its timing, in fact, is a weakness. On commentary, however, its timing works against it: the responses are temporally erratic, arriving too frequently in some passages and too sparsely in others, because its periodic external trigger has no means of judging when a remark is genuinely warranted. This corroborates our central design principle: deciding when to respond must be a native capability of the model, learned and judged internally, rather than a behavior supplied by an external trigger. Doubao therefore prevails on these cases only when its quality advantage is large enough to offset its weaker timing under the equal-weight scoring. Our own losses, in turn, are confined to quality: the model occasionally hallucinates during commentary, a limitation we attribute primarily to its parameter scale and regard as a principal direction for future work. Gemini’s single foothold is time awareness, the closest-contested axis against it, where it ties 40% of cases and wins 10%, holding our win rate to 50%, while on every other scenario it fails to win a single comparison. The reason is specific to this scenario: some time-awareness cases are user-triggered, with the question asked only after the relevant moment has passed, so the real-time demand is low. On these ask-after-the-fact questions, Gemini’s strong underlying model answers well on quality alone. In both cases the baselines close the gap only where timing pressure is lowest, winning on raw answer quality or on questions that no longer require a split-second response, and they fall furthest behind exactly where a reply must land at the right moment.
Case Study.
To make the quantitative gap concrete, we walk through six representative cases; the full side-by-side video comparisons are available in our blog.222https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction Each contrasts JoyAI-VL-Interaction with Doubao’s and Gemini’s in-app video-call assistants on identical input.
In the Fall Detection Alert case (02 Monitoring and alerting in the blog), a person collapses on camera. JoyAI-VL-Interaction raises the alert at the instant of the fall, whereas Doubao reacts four to five seconds later, and Gemini explicitly concedes that it is unable to monitor the scene at all. The lag is not incidental: it is the polling interval of an external trigger surfacing as latency on an event that allows no delay. The Real-time Dart Throw Counting case (05 Real-time counting in the blog) tests sustained, event-locked counting. JoyAI-VL-Interaction increments its count exactly as each dart strikes the board, registering all six throws on time, while Doubao counts only two and with noticeable delay, and Gemini offers a single “let me check” before falling silent. The Street Interview Translation case (01 Real-time translation in the blog) probes continuity, with the task being to translate the interview’s on-screen subtitles in real time rather than its audio. JoyAI-VL-Interaction translates the speech continuously and stays accurate even when Chinese pinyin appears in the on-screen subtitles, whereas both Doubao and Gemini translate only what was visible at the moment the request was issued and then stop, treating an ongoing task as a single turn. The Timed Cooking Scene case (06 Time awareness in the blog) asks the system to signal once a twenty-second interval has elapsed, a direct probe of an internal sense of time. JoyAI-VL-Interaction is off by only one to two seconds; Doubao fails to signal at the mark altogether, and Gemini does so only at around forty seconds, roughly double the target. The Pet Livestream Commentary case (04 Live commentary in the blog) plays a continuous stream of short clips of different pets, requiring the model to keep pace with the changing scene and to recognize and narrate each animal’s emotional state. JoyAI-VL-Interaction narrates each pet as it appears, whereas Doubao cannot keep up, describing only three of thirteen and inaccurately at that, and Gemini comments just once, and only when prompted. The Phone App Delegation case (09 Agent delegation in the blog) illustrates a capability the baselines do not offer at all: when a phone’s app interface appears on screen, the model is asked to produce HTML that renders a similar screen. JoyAI-VL-Interaction recognizes that this exceeds real-time inference, hands the task to the background model, and returns working HTML reproducing the observed interface, all without leaving the live session.
Emergent capabilities.
Two further cases are more striking still, in that they exercise capabilities the model was never trained for. In the Shopping App Guidance case (03 App guidance in the blog), the user pursues a shopping goal while swiping through a phone’s screens. JoyAI-VL-Interaction responds promptly and narrates in step with each swipe, guiding the user to the intended item, whereas neither Doubao nor Gemini reacts proactively or in real time, describing only a few of the screens first shown. Notably, our training data contains no app-interface video of any kind: the ability to follow a changing interface is one the model acquired on its own, generalizing to a domain it never saw. In the Travel Scene Commentary case (04 Live commentary in the blog), the system is asked to narrate once every four seconds. JoyAI-VL-Interaction holds to this cadence throughout and keeps the commentary substantively strong; Doubao narrates only the opening scene for some twenty seconds and ignores the four-second instruction entirely, while Gemini, though asked in Chinese, replies in English and comments only once. The task combines two abilities, timed action and live commentary, that never co-occur in our training data: the model composes them at inference time, an emergent crossing of capabilities rather than a pattern it was shown.
Across these cases the same factor separates JoyAI-VL-Interaction from systems many tens or even hundreds of times its size: timing. By making the decision of when to respond a native, learned capability of the model rather than a behavior imposed by an external trigger, JoyAI-VL-Interaction acts at the moment each event demands, where larger but turn-based or polling-based systems arrive late, stop after a single turn, or never engage. Moreover, its compact size imposes no ceiling on what it can ultimately handle: when a problem genuinely calls for a more powerful model, JoyAI-VL-Interaction does not attempt to answer it inline but delegates it, dispatching the hard subtask to a background model through an asynchronous request and folding the result back in once it returns, all while remaining present with the user in real time. A compact 8B model thus pairs real-time, well-timed presence with on-demand access to heavier reasoning, which we regard as the decisive reason it can outperform far larger models and mature products on the dimension that matters most in the streaming setting.
5.3 Limitations
A fair reading of our comparison has to start with scale. The two video-call assistants we evaluate against are powered by far larger models, Seed 2.0 behind Doubao and Gemini-3.1-flash-live behind Gemini, and both are mature products tuned over time against real users and use cases. JoyAI-VL-Interaction, by comparison, is a compact 8B-scale model. We therefore expect, and do not claim otherwise, that on general turn-based ability these products are stronger than ours: broader world knowledge, a more polished one-on-one chat experience, and greater robustness on complex or rarely seen inputs.
Our claim is narrower, and we think more telling. In the six event-driven scenarios this paper targets, the ones that turn on real-time presence, proactive interaction, and a sense of time, and that already carry rich deployment value, a far smaller model holds a natural advantage simply by being an interaction model rather than a turn-based one. That a compact, open model can outperform far larger and heavily optimized products precisely in this regime is, to us, the point: interactivity is worth scaling as a capability in its own right, and it begins to pay off well before enormous scale.
Both the data and the evaluation behind this release are, by deliberate choice, still at an early stage. On the data side, we have not yet tuned the mixture or scaled it to the volume we ultimately intend, and a further round of cleaning remains; one consequence is already visible in our results, where the comparatively sparse commentary data, together with the model’s compact scale, is the main cause of the occasional hallucination during live narration (§5). The evaluation is similarly preliminary, six scenarios and 58 human-rated cases against two products, rather than the larger, more fine-grained study we ultimately want. We release at this stage regardless, and the reason is the same for both: even with this early data and this limited evaluation, the model already exhibits interaction capabilities we never explicitly trained for, and watching them emerge convinced us, more than any single number could, that interactivity is a direction worth scaling in its own right. That conviction is exactly why we would rather put the model, the recipe, and the system into the community’s hands now than hold them back for a polished data recipe and a larger benchmark: we are eager to scale this direction together rather than alone. A subsequent version will tune the data mixture, scale the corpus, and report a broader and more systematic evaluation; in the interim, it is the data-construction methodology and the approach itself, rather than this particular corpus or benchmark, that we expect others can most readily adapt and extend. These limitations also chart the road ahead: closing the gap in general ability and robustness, widening the set of scenarios, and pushing further on what an interaction model can do.
6 Conclusion
We set out to move large models past the turn-based paradigm, in which a model opens its eyes only when it is addressed, toward genuine streaming interaction, in which the model is present in the world and decides for itself when to act. To that end we release, to our knowledge, the first open vision-driven interaction model, together with everything needed to build on it: the time-aligned data, the training recipe, the 8B model, and a complete, deployable system.
At the center is a single conviction, that interactivity should scale as a capability of the model itself rather than be simulated by a harness around it. Scaling interactivity means scaling three things at once: the model’s freedom to speak on its own when a moment is worth a word, its ability to interact in real time, and its sense of elapsed time. As these grow, a model behaves less like a tool that waits to be queried and more like a participant that is simply present, watching what is happening and responding as a person would. We see this as a direction of scaling in its own right, alongside the scaling of intelligence, and one that brings models closer to being genuinely present in human settings.
Across the streaming scenarios we study, we already see early signs that an interaction model carries natural advantages for real deployment, from monitoring and live narration to step-by-step guidance and companionship. The moment we are reaching for is an everyday one: you come home worn out after a long day, and before you have said anything, a quiet voice notices and offers, “I can see you’re tired; today must have been hard on you.” Presence like that, given unasked, is what an interaction model makes possible and a turn-based one, waiting to be addressed, never can. Much remains open in both the model and the system, and we have released the whole stack openly to lower the barrier and accelerate this shift. Turn-based models, products, and optimizations have been built up over many years, while vision-driven interaction models are only beginning; if this release opens even a first foothold for them among those long-settled peaks, and draws others into building it with us, it will have done its job. We invite the community to explore, with us, what a model that is truly present in the world can become.
References
- Bai et al. [2025] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. https://arxiv.org/abs/2511.21631.
- ByteDance Seed [2026] ByteDance Seed. Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity. Available at ByteDance Seed Model Cards, 2026.
- Chen et al. [2024] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR, 2024.
- Chien et al. [2026] Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, and Alexandre Défossez. Moshirag: Asynchronous knowledge retrieval for full-duplex speech language models, 2026. https://arxiv.org/abs/2604.12928.
- Du et al. [2025] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Xian Shi, Keyu An, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589, 2025.
- Gao et al. [2023] Zhifu Gao et al. Funasr: A fundamental end-to-end speech recognition toolkit. In INTERSPEECH, 2023.
- Google [2026] Google. Gemini 3.1 Flash Live: Making Audio AI More Natural and Reliable. Google Blog (The Keyword), March 2026. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-live/. Posted 2026-03-26. Accessed 2026-06-06.
- Guan et al. [2026] Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, and Xiang Bai. Video streaming thinking: Videollms can watch and think simultaneously. arXiv preprint arXiv:2603.12262, 2026.
- Hou et al. [2026] Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, and Jiaqi Wang. Adacodec: A predictive visual code for video mllms, 2026. https://arxiv.org/abs/2606.02569.
- Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. https://arxiv.org/abs/2309.06180.
- Lab [2026] Thinking Machines Lab. Interaction models: A scalable approach to human-ai collaboration. Thinking Machines Lab: Connectionism, May 2026. 10.64434/tml.20260511. https://thinkingmachines.ai/blog/interaction-models/.
- Li et al. [2025] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How far is your video-llms from real-world online video understanding?, 2025. https://arxiv.org/abs/2501.05510.
- Lin et al. [2024] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding, 2024. https://arxiv.org/abs/2411.03628.
- Liu et al. [2026] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization, 2026. https://arxiv.org/abs/2601.05242.
- Lu and Lab [2025] Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.
- Nous Research [2026] Nous Research. Hermes agent: A self-improving open-source ai agent. https://github.com/NousResearch/hermes-agent, 2026. GitHub repository.
- OpenAI [2026] OpenAI. Advancing voice intelligence with new models in the API (GPT-Realtime-2). OpenAI Blog, May 2026. Announced 2026-05-07. https://openai.com/index/advancing-voice-intelligence/. Accessed 2026-06-02.
- OpenClaw Foundation [2026] OpenClaw Foundation. Openclaw: An open-source autonomous ai assistant. https://github.com/openclaw/openclaw, 2026. GitHub repository.
- Qian et al. [2025a] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint arXiv:2501.03218, 2025a.
- Qian et al. [2025b] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:119336–119360, 2025b.
- Qin et al. [2026] Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Dingyu Yao, Zheng Lin, Peng Fu, Nan Duan, and Jiaqi Wang. Easyvideor1: Easier rl for video understanding, 2026. https://arxiv.org/abs/2604.16893.
- Qwen Team [2026] Qwen Team. Qwen3.5-omni technical report, 2026. https://arxiv.org/abs/2604.15804.
- Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. https://arxiv.org/abs/2402.03300.
- Volcano Engine (2026) [ByteDance] Volcano Engine (ByteDance). Real-time conversational AI: Video and image understanding. Volcano Engine Documentation, 2026. https://www.volcengine.com/docs/6348/1408245. Accessed 2026-06-02.
- Wang et al. [2026] Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, and Linfeng Zhang. Accelerating streaming video large language models via hierarchical token compression, 2026. https://arxiv.org/abs/2512.00891.
- Wang et al. [2024] Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format, 2024. https://arxiv.org/abs/2411.17991.
- Xia et al. [2025] Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, and Kaiyang Zhou. Streaming video instruction tuning, 2025. https://arxiv.org/abs/2512.21334.
- Xu et al. [2025] Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams, 2025. https://arxiv.org/abs/2510.09608.
- Yan et al. [2026] Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Hao Liao, Xing Xie, Tao Jin, and Jianxun Lian. Proact-vl: A proactive videollm for real-time ai companions, 2026. https://arxiv.org/abs/2603.03447.
- Yang et al. [2025] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. https://arxiv.org/abs/2505.09388.
- Yao et al. [2026] Dingyu Yao, Shuhuan Gu, Qingyi Si, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Naibin Gu, Zheng Lin, Weiping Wang, Nan Duan, and Jiaqi Wang. Harnessing streaming video in the wild, 2026. https://arxiv.org/abs/2606.08615.
- Yao et al. [2025] Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 10807–10816, 2025.
- ydyhello [2026] ydyhello. Awesome-VLM-Streaming-Video: A Curated Collection of Papers and Open-Source Code on Vision-Language Models for Streaming Video. GitHub repository, 2026. https://github.com/ydyhello/Awesome-VLM-Streaming-Video. Accessed 2026-06-06.
- Zeng et al. [2026] Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, and Limin Wang. Streamforest: Efficient online video understanding with persistent event memory. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. https://openreview.net/forum?id=9loSPaBwGO.
- Zhang et al. [2026] Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng, and Xipeng Qiu. Hermes: Kv cache as hierarchical memory for efficient streaming video understanding, 2026. https://arxiv.org/abs/2601.14724.
- Zheng et al. [2024] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. https://arxiv.org/abs/2312.07104.