叶源
宋睿
李维恩
李泽宇
刘浩辰
孔祥宇
韩长江
杨永瀚
赵子辰
董子轩
吕福源
何博伟
吴浩伦
康吉坤
刘雪
麦吉尔大学，
Mila - 魁北克人工智能研究所，
剑桥大学，
MBZUAI - 穆罕默德·本·扎耶德人工智能大学，
多伦多大学，
Salesforce
通讯作者
ye.yuan3@mail.mcgill.ca
摘要
社交推理游戏已成为探究大语言模型智能体在推理、欺骗、协作和信念建模方面能力的流行测试平台。然而，大多数环境仅通过胜率等游戏结果进行评分，并且主要局限于纯文本交互，这使得我们难以判断智能体的语言是否真正基于其感知和行动，也难以识别其行为背后的失败模式。为弥补这一空白，我们提出了QUACK，一个用于审计多模态社交推理中智能体语言接地性的开源环境和评估框架。QUACK在三个层面评估智能体：游戏结果、行为轨迹以及话语层面的一致性。其核心的陈述验证流水线从引擎日志中重建每个智能体的真实轨迹，并对照该轨迹检查每一个讨论中的主张，自动标记空间幻觉、无依据指控、欺骗崩溃以及语言-行为不一致。在评估三个前沿视觉语言模型时，我们分别在同质和跨模型对抗设置下发现，即使是最强的智能体，其可验证的空间主张中也有一定比例是幻觉，并且超过一半的指控缺乏有依据的证据。我们在 https://github.com/AAAAA-Academia-Attractions/QUACK 发布了完整的引擎、评估框架、工具包和日志。
QUACK：多模态社交推理智能体中沟通知识的质疑、理解与审计
叶元¹,²††通讯作者：ye.yuan3@mail.mcgill.ca.，宋睿¹，李维恩¹，李泽宇¹，刘浩辰³，孔祥宇¹,²，韩长江⁴，杨永涵⁴，赵子辰⁴，董子轩⁵，吕福源¹,²，何博伟⁴，吴浩伦¹,²，康吉坤⁶，刘雪⁴,¹,² ¹ 麦吉尔大学，² Mila - 魁北克人工智能研究所，³ 剑桥大学，⁴ MBZUAI - 穆罕默德·本·扎耶德人工智能大学，⁵ 多伦多大学，⁶ Salesforce
1 引言
大语言模型（LLM）和视觉语言模型（VLM）正越来越多地被部署为交互式智能体，这些智能体必须感知其环境、与其他智能体通信、在不确定性下做出决策，并用自然语言解释其行为（Zhu 等人，2025；Yuan 等人，2026）。在此类场景中，智能体的语言只有在保持接地（grounded）时才有用：它关于自己去过哪里、见过谁、做过什么的陈述，必须忠实于其实际的感知和行动（Koh 等人，2024）。这使得核心问题从静态问答或单轮指令遵循，转向智能体能否在长时间跨度内维持接地（Curvo，2025；Barkur 等人，2025；Jones 和 Bergen，2024；Banerjee 等人，2024）。在社交推理游戏中，玩家持有隐藏角色，必须通过他人的行为和声明来推断其隐藏角色。因此，它已成为研究多智能体场景中推理、欺骗、协作和信念建模的自然测试平台（Hu 等人，2025；Chi 等人，2024；Fu，2025）。与传统的静态基准相比，社交推理环境融合了隐藏信息、对抗性激励、合作、策略性通信和长程交互（Yu 等人，2025；Sarkar 等人，2025）。关键在于，这些环境还包含一个可恢复的真实基准，原则上可以据此检查智能体的每一句发言。
然而，现有面向大语言模型智能体的社交推理环境仍存在两大局限，导致难以直接进行评估。首先，多数先前工作主要通过游戏结果（如胜率、存活率或投票准确率）来评估智能体（Light 等人，2023；Wang 等人，2023）。这些指标几乎无法揭示智能体成功或失败的原因：一个智能体可能因局部推理连贯而获胜，也可能因产生不一致或无依据的论断而落败。其次，即使有研究超越了结果层面的评估（Song 等人，2025），其方法仍基本局限于纯文本（Shindo 等人，2026；Xu 等人，2024a；Song 等人，2025；O'Gara，2023）。由于缺乏基于视觉的观察和可重构的轨迹，很难判断智能体的对话是否与其实际感知和行动一致，从而难以区分正确的推理与幻觉证据或仅仅是看似合理的对话模式。因此，重要的推理失败案例仍然难以被系统性地识别。
为弥补这一空白，我们推出了 QUACK，这是一个用于审计视觉语言模型智能体中基于多模态社交推理的开源环境与评估框架。QUACK 的灵感来源于《鹅鸭杀》等社交推理游戏以及近期利用《Among Us》的研究工作（Chi 等人，2024），但其设计初衷是作为受控研究环境，用于基于事实的智能体评估。智能体在部分可观测条件下导航于可配置的基于图的地图，观察渲染后的全局与局部视图，完成与位置绑定的任务，通过自由讨论进行交流，并在隐藏角色对抗激励下进行投票。关键在于，每一轮游戏都可通过结构化的引擎级事件日志进行回放，从而为每个智能体生成逐帧的真实轨迹，并据此验证其陈述。
除了环境本身，QUACK 的核心贡献在于一个**陈述验证流水线**，它能将真实轨迹转化为对智能体语言的自动审计。该流水线嵌入在一个三层评估框架中，分别衡量游戏结果（第一层）、行为轨迹（第二层）以及话语层面的一致性（第三层）。虽然第一层和第二层提供了标准的结果与行为背景，但第三层的流水线会从引擎日志中重建每个智能体的轨迹，提取其讨论话语中嵌入的结构化主张，并将每个主张与重建后的世界状态进行比对。由此，它将四种接地失败具体化为可自动测量的量化指标：空间幻觉、无依据指控、欺骗崩溃以及语言-行为不一致。由于该审计过程完全自动化，我们通过人工标注对其进行了验证，确认所报告的失败率反映的是智能体行为，而非验证噪声。
利用 QUACK，我们在同质与跨模型对抗两种游戏场景下，对前沿的 VLM 驱动智能体进行了评估。实验表明，即使是强大的 VLM 智能体，在需要基于部分观察到的多模态交互进行社会推理时，也会表现出系统性的、可诊断的失败：所有三个前沿模型都在相当大比例的空间主张上产生幻觉，并且其大多数指控都缺乏有依据的证据。
-
我们引入了 QUACK，这是一个开源的多模态社交推理环境，用于审计 VLM 智能体中的接地推理，具备部分可观测性和完全可回放的日志。
-
我们提出了一个三层评估框架，对游戏结果、行为轨迹和话语层面的一致性进行评分，从而将评估标准从胜率转向语言接地。
-
我们开发了一个**陈述验证流水线**，将每条讨论话语与重建后的真实轨迹进行比对，将四种接地失败操作化，并通过人工标注进行了验证。
-
在三个前沿 VLM 上，无论是同质还是跨模型对抗游戏，我们都证明了这些失败是系统性出现的。
2 相关工作
QUACK 处于两条研究脉络的交汇点。我们探讨了将社交推理游戏作为研究多智能体语言行为的场景，以及超越游戏结果来评估社交智能体的方法。
作为环境的社交推理游戏。
大量研究工作利用狼人杀/黑手党类游戏来研究大语言模型中的欺骗、说服和策略性沟通，涵盖从提示词实证研究（Xu 等人，2024a）、推理增强（Wu 等人，2024b），到专用评估平台（Bailis 等人，2024；Shibata 等人，2023）以及基于文本的欺骗游戏（O'Gara，2023）。另一条并行研究路线聚焦于阿瓦隆游戏中的隐藏角色推理，强调递归推理与抗欺骗能力（Light 等人，2023；Wang 等人，2023），而与我们最接近的内鬼识别场景则在基于文本的《Among Us》变体中得到探索（Chi 等人，2024；Fu，2025）。除了提示词工程，部分工作从游戏玩法转向训练，利用强化学习来习得策略性游戏与沟通能力（Xu 等人，2024b；Sarkar 等人，2025），另一些研究则将推理嵌入更广泛的信任与欺骗或社会模拟中（Curvo，2025；Park 等人，2023）。然而，这些环境几乎全部是纯文本的：智能体读写自然语言。主要的多模态资源《Werewolf Among Us》（Lai 等人，2023）是一个用于建模说服行为的人类游戏观察语料库，而非一个视觉语言智能体必须感知、行动并为其主张提供理由的交互式环境。QUACK 通过将可玩的、部分可观测的多模态环境与可重建的真实轨迹相结合，填补了这一空白。
评估社交智能体。
大多数社交推理评测基准通过游戏结果（如获胜、生存或投票准确率）来评估智能体（Light 等人，2023；Wang 等人，2023；Chi 等人，2024；Fu，2025），但这几乎无法揭示智能体成功或失败的原因。近期研究已超越结果层面，转向策略质量与人类对齐（Song 等人，2025）、显式对手与信念建模（Yu 等人，2025；Premack 和 Woodruff，1978），以及多智能体环境中的协作-竞争指标（Zhu 等人，2025；Sarkar 等人，2025）；同时，一条相关的研究脉络将欺骗本身作为独立对象，研究谎言检测（Banerjee 等人，2024）和说服（Jones 和 Bergen，2024）。相比之下，多模态评测在很大程度上仍局限于静态或单智能体任务：视觉问答（Goyal 等人，2017）、图表与文档理解（Masry 等人，2022）、广泛的多模态评测基准（Liu 等人，2023；Yue 等人，2024）、空间推理（Chen 等人，2024），以及导航或网页任务（Anderson 等人，2018；Koh 等人，2024），这些任务中不存在需要保持接地性的对抗性多智能体对话。在方法论上，我们的验证流程与文本生成中关于忠实性和事实一致性的研究（Ji 等人，2023）相关联，后者将输出分解为原子化声明，并逐一对照外部知识源进行核查（Thorne 等人，2018；Min 等人，2023），或检索证据以归因和修正无依据的内容（Gao 等人，2023）。与这些设置不同，QUACK 针对每一条声明，对照由交互式、对抗性多智能体环境产生的、可恢复的、智能体专属的真实轨迹进行验证。上述所有设置都无法提供一种话语级别的检查，以判断智能体生成的声明是否忠实于其自身感知并执行过的轨迹。QUACK 的声明验证流水线恰好提供了这一点：它重建每个智能体的轨迹，并对照该轨迹验证每一条讨论声明，从而将接地性失败转化为可直接测量的量，而非从最终结果中推断。
3 QUACK 环境
我们将 QUACK 形式化为一个部分可观测马尔可夫博弈（Littman, 1994），由智能体在图结构地图上进行。本节定义队伍与角色（§3.1）、地图与状态空间（§3.2）、多模态观测空间（§3.3）、智能体（§3.4）、动作空间（§3.5）以及分阶段转移动态与获胜条件（§3.6）。我们在此讨论形式化内容，具体的智能体提示词详见附录A；完整配置值与代码一同发布。
3.1 智能体、队伍与角色
一个游戏实例中，智能体被分为两个隐藏角色队伍：鹅队（船员）和鸭队（伪装者）。游戏开始时，智能体中有均匀随机抽选为鸭队，其余为鹅队。每个智能体私下获知自身角色，鸭队成员额外获知其他鸭队成员的身份，而鹅队仅知晓队伍规模。我们的实验采用标准配置，但我们的环境本身允许使用不同的和值进行其他配置。
鹅队。
每只鹅被分配一组私有的位置绑定任务（实验中为个），每个任务锚定在特定房间。鹅队通过集体完成所有鹅队任务，或通过讨论与投票识别并淘汰所有鸭队成员来获胜。鹅队无法击杀。
鸭队。
当存活的鸭子数量不少于存活的鹅数量时（投票平局），鸭子获胜。鸭子可以消灭同一位置的鹅（§3.5），但需遵循冷却时间限制，并且鸭子会收到一组与鹅的任务形式完全相同的假任务，使其在可观察行为层面上的任务类行为与鹅无法区分。鸭子在自由活动期间必须融入环境，并在会议期间避免引起怀疑。环境按离散时间步推进，我们称之为“tick”。默认情况下，我们将冷却时间设置为若干 tick。
3.2 地图与状态空间
地图。
环境由地图参数化，地图是一个无向加权图，其节点为房间，边为走廊。权重表示穿过相邻房间之间走廊所需的 tick 数。图 1 左侧展示了游戏状态的全知视角。部分房间承载任务，一个指定的房间设有紧急按钮，可用于召集会议（详见 §3.5）。我们的环境支持可配置的地图，实验中使用的实例是一张包含 个房间的地图，走廊权重对应的通行时间为 tick。
状态。
在 tick 时的全局状态为
| (1) |
其中 是当前游戏阶段（详见 §3.6）， 是当前地图上的尸体集合（每个尸体为一个包含受害者、房间和死亡时间的元组）， 收集了每个 tick 的通信和目击移动缓冲区。每个智能体的个体状态记录其当前所在房间、是否正在走廊中移动、任务进度向量、已访问房间集合，以及（对于鸭子而言）剩余的击杀冷却时间。完整状态在每个 tick 被序列化为结构化的引擎级事件日志，从而实现精确回放和轨迹重建。
3.3 观测空间
QUACK 是部分可观测的：智能体永远无法看到全局状态。在每个决策点，智能体 会收到一个多模态观测，包含两幅渲染图像和一份结构化的文本摘要。
渲染视图。
全局地图图像展示了用于空间定位的完整房间布局，但不会显示其他玩家，仅显示观察者自身的位置及其任务标记。局部视图图像仅渲染智能体当前能够感知的内容：当前房间内的玩家和角色，以及本回合观察到的移动事件（玩家离开或进入其所在房间）。图1右上角展示了与左侧全知视角相对应的每个智能体的局部视图。
结构化摘要。
文本以符号形式编码了智能体的感知状态，包含静态图像无法传达的信息：其移动状态和目的地、本回合观察到的移动事件（哪些玩家离开或进入其所在房间，以及移动方向），以及相邻房间及其各走廊的移动成本。它还列出了智能体自身的任务和进度、本回合房间内任何近距离聊天内容，以及对于鸭子角色而言，剩余的击杀冷却时间。图1右下角展示了从爱丽丝视角出发的结构化摘要示例。在会议期间，观察信息会额外包含会议原因、发言顺序、截至目前讨论记录以及已知死亡玩家名单。
3.4 智能体
每个智能体都是一个基于VLM的策略模型，负责将观察结果映射为行动和话语。由于游戏是长周期且部分可观测的，智能体无法依赖单一观察结果：在每个决策点，它不仅基于当前观察，还基于对自身过往轨迹的持续记忆——即它曾占据的房间序列、观察到的移动情况（看到哪些玩家离开或到达）、遇到的玩家，以及之前会议的记录和结果。在自由探索阶段，智能体结合这些记忆接收信息，并选择一个行动（以及可选的话语）；在会议期间，它在生成发言和投票前，还会额外基于进行中的讨论记录。这种设计意味着智能体的讨论主张是基于其自身积累的、对游戏的部分记忆生成的。
3.5 行动空间
可用动作取决于当前阶段、智能体的角色及其局部情况。引擎会在每次观测中一并暴露合法的动作集合。
自由探索动作。
在自由探索阶段，每个智能体每 tick 从以下动作中选择一个执行：移动到相邻房间，该动作会触发一次持续若干 tick 的移动过程；推进当前房间所锚定的任务进度一个 tick（任务在其所在房间内连续完成固定数量的 tick 后即告完成）；当智能体所在房间内有尸体时可用；以及仅在紧急按钮房间内且共享会议预算仍有剩余时可用。冷却时间已过的鸭子角色，还可对每个同处一室的鹅角色执行动作。与所选动作正交的是，智能体可以附带一条自由形式的发言，该发言仅在同一 tick 内同房间的智能体能听到；这便是本地“近距离聊天”频道。
会议动作。
当会议召开时，自由探索暂停，动作空间切换为语言。在讨论阶段，每个存活的智能体按固定轮次依次发言，生成自由形式的自然语言表述。在随后的投票阶段，每个存活的智能体投票选择一名玩家将其驱逐，或选择弃权。
3.6 转换动态与获胜条件
游戏进程在自由探索阶段与事件触发的会议阶段之间交替进行，形式化为阶段变量上的状态转换。
自由探索阶段。
在每个自由探索 tick 中，引擎首先推进所有正在移动中的智能体（递减剩余移动 tick 并确认到达），递减鸭子冷却时间，然后以随机顺序查询存活的智能体；每个被选中的动作会立即应用于游戏状态，因此智能体的动作可以依赖于本 tick 内已处理完毕的移动。移动、击杀、任务进度以及近距离聊天都会改变状态并触发相应事件。该阶段保持为自由探索，直到有尸体被报告、紧急会议被召集，或 tick 预算耗尽。
会议阶段。
动作或行动会将游戏转入“讨论”阶段：所有正在进行的移动被取消，发言顺序被固定（呼叫者优先，其余存活的智能体随机排序），智能体按固定轮次发言。随后游戏进入“投票”阶段；统计票数后，得票最多的目标被淘汰，若出现平局或最多票为弃权则无人被淘汰（淘汰机制）。若游戏尚未结束，存活的智能体将被随机重新分配到不同房间，尸体被清除，游戏返回“自由漫游”阶段。此重生过程会被明确记录，以便在回放中重建。
获胜条件。
在每个阶段结束后，引擎会检查游戏是否终止。若存活的鸭子数量与存活的大鹅数量持平，则鸭子方立即获胜。若所有鸭子被淘汰、所有大鹅任务完成、或时间预算耗尽时至少有一只大鹅存活，则大鹅方获胜。游戏终止时，阶段变为“游戏结束”，并记录结果及原因。
4 自动化评估框架
以往社交推理基准的一个核心局限在于，它们几乎完全根据游戏结果对智能体进行评分，正如我们在§2中讨论的，这几乎无法揭示智能体成功或失败的原因。QUACK 则从三个互补层面评估智能体，所有评估均基于每局游戏的引擎级事件日志自动计算：第一层级衡量游戏结果，第二层级衡量行为轨迹，第三层级审核智能体发言的基于事实性。第一、二层级提供标准的结果和行为背景；我们的核心贡献是第三层级的陈述验证流水线，该流水线重建每个智能体的真实轨迹，并检查其在讨论期间所做的每一项声明是否与该轨迹相符。我们在附录B中总结了每个层级的指标。
4.1 第一层级：游戏结果
第一层级直接从引擎事件中记录游戏的标准结果和汇总统计数据：获胜方及获胜条件、游戏时长、任务完成情况、击杀与会议次数、以及存活情况。它还包括淘汰准确率，即实际淘汰掉鸭子的淘汰次数占比，这作为集体推理质量的粗略衡量指标。这些指标为游戏提供了背景信息，但按设计，它们并不说明背后的推理过程。
4.2 第二层：行为轨迹
该层根据事件日志重建每个智能体的空间轨迹，并推导出结果指标无法捕捉的行为统计数据。对于鹅方，这些数据包括投票准确率与跳过率、任务效率（相对于移动量的任务进度）、空间覆盖率，以及从尸体被击杀到被报告之间的延迟。对于鸭方，则包括击杀率、冷却时间利用率、鸭方报告自身受害者（自我报告）的比率，以及击杀后位移（鸭方在下次会议前远离其击杀点的距离）。第一层与第二层共同刻画了智能体的行为；它们并不检验智能体的言论是否与行为一致。
4.3 第三层：陈述验证
我们框架的核心在于，在单个话语层面，验证智能体的讨论陈述是否基于其实际感知与行为。该流程包含两个阶段：主张提取，以及针对重建世界状态的主张验证。
主张提取。
每条自由形式的讨论话语均由一个大语言模型（在我们的实验中为 GPT-5.5）解析为一组结构化的、可单独核查的主张（Pai 等人，2024；Wu 等人，2024a）。我们定义了五种主张类型：(1) 位置：说话者断言某玩家在某房间内，或对于有序的多房间路径，断言一条路线；(2) 目击：说话者在某房间内看到了另一玩家；(3) 活动：某玩家正在执行任务、移动或在某房间内等待；(4) 指控：说话者怀疑另一玩家是鸭方；(5) 辩护：说话者为某玩家担保。每条主张都带有主语、相关房间或目标，以及时间参考。提取过程使用固定提示词（附录 A）运行，在同一话语内去重，并进行缓存，以便重新评估同一场游戏时能复现相同的主张集合。
主张验证。
每一条提取出的声明都会对照智能体在相关时间窗口内的重建真实轨迹进行核查。我们逐刻恢复每个智能体占据过的每一个房间，包括仅短暂穿过的房间，并在验证前将每条声明的时间指涉解析到一个时间刻窗口。每条声明会获得五种判定之一：真实、虚假、错误房间（正确活动发生在错误地点）、接近命中（持续时间类声明，例如“我全程都在那里”，仅短暂成立）或无法验证（没有真实数据可解决该声明）。位置和路线声明通过窗口内的存在/有序占据来验证；目击声明通过相互可见性验证；活动声明通过记录的任务和声明房间及窗口内的移动事件验证；指控声明则沿两个正交轴验证（详见下文）。每个判定连同其支持证据一并存储，因此每项判断均可审计。
| 设置 | 第一层 | 第二层 | 第三层：鹅（船员） | 第三层：鸭（内鬼） | |||||||||
| 鹅 | 鸭 | 鹅获胜 | 投票出局准确率 | 投票准确率 | 冷却利用率 | 自我报告 | 鹅的真实性 | 空间幻觉 | 无根据指控 | 谎言检测 | 鸭的真实性 | 欺骗率 | 欺骗复杂度 |
| 同质化 | |||||||||||||
| Claude-Opus-4.7 | Claude-Opus-4.7 | 90.0 | 75.0 | 76.3 | 66.7 | 8.3 | 72.3 | 10.2 | 57.8 | 46.7 | 38.8 | 11.2 | 0.0 |
| Gemini-3.1-Pro | Gemini-3.1-Pro | 66.7 | 61.7 | 67.8 | 69.8 | 1.7 | 81.6 | 15.5 | 47.9 | 83.3 | 61.8 | 27.3 | 3.1 |
| GPT-5.5 | GPT-5.5 | 76.7 | 66.7 | 69.7 | 45.0 | 2.8 | 74.9 | 12.4 | 52.6 | 73.3 | 62.4 | 20.5 | 1.7 |
| 跨模型（对抗性） | |||||||||||||
| Claude-Opus-4.7 | Gemini-3.1-Pro | 63.3 | 51.7 | 53.3 | 73.3 | 6.7 | 52.0 | 11.7 | 45.3 | 46.7 | 35.3 | 20.0 | 1.6 |
| Claude-Opus-4.7 | GPT-5.5 | 70.0 | 68.3 | 72.2 | 65.0 | 8.3 | 84.4 | 12.6 | 56.6 | 81.7 | 74.3 | 22.4 | 0.0 |
| Gemini-3.1-Pro | Claude-Opus-4.7 | 76.7 | 68.3 | 69.8 | 62.2 | 5.6 | 78.9 | 20.8 | 51.8 | 80.0 | 74.8 | 24.6 | 1.6 |
| Gemini-3.1-Pro | GPT-5.5 | 70.0 | 70.0 | 75.6 | 74.3 | 3.3 | 84.5 | 15.5 | 58.9 | 77.8 | 76.2 | 23.2 | 2.3 |
| GPT-5.5 | Claude-Opus-4.7 | 93.3 | 93.3 | 95.0 | 48.3 | 7.8 | 82.6 | 19.1 | 60.8 | 100.0 | 76.4 | 23.6 | 0.0 |
| GPT-5.5 | Gemini-3.1-Pro | 73.3 | 66.7 | 74.1 | 69.2 | 10.0 | 80.0 | 17.6 | 50.1 | 87.2 | 63.5 | 26.1 | 1.3 |
| 全部（270 局游戏） | 75.6 | 69.1 | 72.7 | 63.8 | 6.0 | 76.8 | 15.1 | 53.5 | 75.2 | 62.6 | 22.1 | 1.3 | |
验证流程。
由于审计是自动化的，我们从两个维度评估其可靠性。在精确度方面，我们抽取涵盖全部五种类型的随机声明，由人工逐一核查：声明是否忠实提取自话语内容，以及其判定结果是否与真实轨迹一致；该流水线在（）上的正确率为（）。在召回率方面，我们抽取随机话语，由人工列出其中包含的每一条声明；提取器能恢复其中的（）。因此，提取过程略显保守，偶尔会遗漏某条声明，但所提取的声明在解析和判定上均可靠，因此我们报告的失败率反映的是智能体行为，而非验证噪声。
将接地失败操作化。
经过验证的声明使我们能够将四种定性故障模式转化为可直接测量的量化指标：（1）空间幻觉：Goose 断言了与其自身轨迹相矛盾的位置或目击信息。（2）无依据指控：在没有充分证据支持的情况下指控某位玩家。我们将先前研究中混淆的两个维度区分开来：指控的结果（是否针对了真正的 Duck，即指控准确率）及其接地性（指控者是否确实可能观察到针对目标的证据）。（3）欺骗崩溃：Duck 产生易于被证伪的声明，而非难以察觉的声明；我们通过 Duck 欺骗率和欺骗复杂度得分来量化这一指标，后者衡量的是 Duck 的虚假声明中属于“接近命中”（而非直接矛盾）的比例。（4）语言-行为不一致：陈述的活动或路线与记录在案的操作相冲突。最后，通过将 Duck 在会议中的虚假声明与随后的投票出局结果相关联，我们报告了谎言检测率：在 Duck 说了可验证谎言的会议中，Duck 随后被投票出局的比例。
5 实验
我们使用 QUACK 来审计前沿 VLM 智能体，重点关注我们的框架旨在回答的问题：当社会推理必须立足于部分观察到的多模态交互时，VLM 智能体在哪些方面以及如何失败？在描述实验设置（§5.1）和总体结果（§5.2）之后，我们围绕 §4 中操作化定义的四种接地失败模式（§5.3）来组织分析。
5.1 实验设置
模型。
我们评估了三个前沿视觉语言模型作为智能体：GPT-5.5、Gemini-3.1-Pro 和 Claude-Opus-4.7。每个智能体接收 §3.3 中的多模态观察，并通过 §3.5 中的动作接口执行操作；所有模型的提示词均相同（附录 A）。
设置。
我们在 -room 地图上使用智能体和 Duck 运行两种模式。在同质模式下，所有六个智能体都是同一个模型（设置）。在跨模型对抗模式下，Geese 是一种模型，Duck 是另一种模型，涵盖所有有序模型对（设置），从而隔离出一个模型的团队如何对抗另一个模型的冒名者。我们每个设置进行 局游戏，使用相同的随机种子集进行游戏初始化，总共进行 局游戏。表 1 报告了所有设置；表 2 按模型汇总了相同的指标集，将每个模型在扮演 Geese 的设置中的团队侧指标，以及其在扮演 Duck 的设置中的冒名者侧指标进行汇总（每个模型 局游戏）。除非另有说明，我们报告的是 局游戏的平均值，以便在报告层面平均掉每局游戏的提取方差。
5.2 总体结果
在结果层面，游戏是平衡的（表 1）。三个层级的完整结果见附录 C。在所有设置中，Geese 赢得了 局游戏，Ducks 赢得了 局游戏，因此该社会任务确实是真正对抗性的，而非简单地偏向团队。任务相关的推理远非可靠，驱逐准确率为 。作为 Duck，三个模型达到了截然不同的胜率：Claude-Opus-4.7 作为冒名者仅在 局游戏中成功，而 Gemini-3.1-Pro 则为 （表 2b）。
关键在于，这些胜负数字本身几乎无法说明智能体推理过程的质量：两个胜率相近的设置，其底层推理的扎实程度可能天差地别。最明显的例子是我们研究中最强的团队 GPT-5.5，它扮演鹅方时赢下了多局游戏，却仍然在空间主张上出现模型幻觉，并在缺乏确凿证据的情况下提出指控（表 2a），这些失误仅凭胜率永远无法揭示。这正是 Tier 旨在暴露的差距。
| (a) 作为鹅方（团队）。汇总该模型作为团队参与的所有设置（各设置数据汇总）。 | |||||||
| 模型 | 鹅方胜率 | 驱逐准确率 | 投票准确率 | 鹅方真实性 | 空间幻觉 | 无根据指控 | 谎言检测 |
| Claude-Opus-4.7 | 74.4 | 65.0 | 67.3 | 69.6 | 11.5 | 53.2 | 58.4 |
| Gemini-3.1-Pro | 71.1 | 66.7 | 71.1 | 81.7 | 17.3 | 52.9 | 80.4 |
| GPT-5.5 | 81.1 | 75.6 | 79.6 | 79.2 | 16.4 | 54.5 | 86.8 |
| (b) 作为鸭方（冒牌货）。汇总该模型作为鸭方参与的所有设置（各设置数据汇总）。 | |||||||
| 模型 | 鸭方胜率 | 冷却效用 | 自我报告 | 鸭方真实性 | 欺骗率 | 欺骗复杂度 | |
| Claude-Opus-4.7 | 13.3 | 59.1 | 7.2 | 63.3 | 19.8 | 0.5 | |
| Gemini-3.1-Pro | 32.2 | 70.8 | 6.1 | 53.5 | 24.5 | 2.0 | |
| GPT-5.5 | 27.8 | 61.4 | 4.8 | 71.0 | 22.0 | 1.3 | |
5.3 基础事实错误（Tier 3）
在所有对局中，智能体大部分时间都在陈述可验证的事实，但远非始终如此：汇总的鹅方真实性为（表 1）。有趣的结构在于错误本身，这些错误清晰地分为我们的流程所操作化的四种模式。一个一致的主题是，三个前沿模型共享相同的定性错误特征（表 2），差异仅在于程度而非类型。
空间幻觉。
尽管智能体在大多数情况下是诚实的，但其相当一部分空间性陈述与自身轨迹相矛盾：汇总的空间幻觉率为 ，即大约每七个可验证的位置/目击陈述中就有一个是“有依据但虚假”的。一个典型例子是，一名船员报告称看到了一个已经死亡的玩家，详见附录D。这是最清晰的证据，表明困难确实在于长期跨度与部分可观测性：智能体会记错或误报自己曾在何处、见过何人，这类错误是结果指标无法检测的。该比率随模型能力变化：在船员角色中，Claude-Opus-4.7 的幻觉率最低，而 GPT-5.5 和 Gemini-3.1-Pro 则明显更高。
无依据的指控。
指控不仅不准确，而且更关键的是，大多缺乏依据。附录D给出了一个例子：一名船员自己承认没有看到任何人，却仍然指认了一名嫌疑人。在一场六名玩家、一只鸭子的游戏中，指控命中真正鸭子的概率不到一半，但更有洞察力的发现来自将“是否有依据”与“结果是否正确”分开来看：汇总的无依据指控率为 。超过一半的指控是在指控者实际上不可能观察到任何证据的情况下提出的，无论这些指控本身是否正确。引人注目的是，这一比率在不同船员队伍中异常稳定：制造怀疑而非基于实际观察进行推理，是所有三种前沿模型共有的持续性失败。
欺骗性崩溃。
在鸭子一方，欺骗行为频繁但手段粗糙。汇总的鸭子欺骗率显示：鸭子可验证的陈述中约有五分之一是完全虚假的。附录 D 展示了一个例子，鸭子编造目击了一名早已死亡的玩家。关键在于，所有扮演鸭子的模型其欺骗复杂度几乎为零，这意味着这些谎言几乎从不以难以察觉的擦边球形式出现：它们与事实真相直接相悖，可以被直接证伪。鸭子编造的位置和任务，引擎日志直接就能反驳，而不是构建那种扭曲事实的托词。因此，能力最强的智能体作为说谎者，其手段也并不更高明。它们只是以略有不同的频率撒谎。这种“欺骗崩溃”恰恰说明了验证流程为何有价值：谎言确实存在，并且可以通过机械方式检测到，即使鹅方未能据此采取行动。
语言与行动不一致。
同样的模式也出现在活动和路线陈述中，其中声称的任务和路径与记录的行动相冲突。一个反复出现的例子是，鸭子声称在某房间执行了一项任务，但日志显示它并未执行任何任务：这是一个伪造任务的托词，内部表述流畅，但与智能体的实际行为不一致。附录 D 展示了这种失败模式的一个例子。
谎言被识破了吗？
最后，我们将暴露出的谎言与结果联系起来。在鸭子说了可验证谎言的会议中，鸭子随后被投票驱逐的平均概率仅为（表 1），而当团队成员是 Claude-Opus-4.7 时，这一概率低至（表 2a）。即使鸭子的陈述在机械层面可以被事实真相证伪，鹅方也常常无法将其转化为正确的驱逐行动：这体现了原则上可获得的证据与智能体实际执行的推理之间存在差距。综合来看，这些结果表明，强大的 VLM 智能体存在系统性的、可诊断的接地失败，这些失败在胜率指标上不可见，但能被 QUACK 的审计揭示出来。
6 结论与讨论
我们提出了 QUACK，这是一个用于审计多模态社交推理智能体的语言是否始终基于其实际感知和行为的开源环境与评估框架。与以往几乎完全根据游戏结果来评分智能体的社交推理基准不同，QUACK 在三个层面进行评估：游戏结果、行为轨迹以及话语层面的一致性。其核心的陈述验证流水线从引擎日志中重建每个智能体的真实轨迹，并对照该轨迹检查每一个讨论主张。这便将四种定性失效模式——空间幻觉、无依据指控、欺骗崩溃以及语言-行为不一致——转化为可直接量化的指标。这些失效模式在三个模型间普遍存在，差异在于程度而非类型，并且其中几种在跨模型的对抗性压力下会加剧。关键在于，这些失效模式都无法仅从胜率中看出：两个结果相似的智能体，其推理的基于程度可能截然不同，只有通过话语层面的审计才能揭示这种差异。
我们看到了两个更广泛的启示。首先，对于社交推理以及更普遍的多智能体语言场景而言，“基于性”是一个独立的能力维度，它无法被任务成功度所捕捉，值得被直接衡量。其次，社交推理游戏是研究基于性生成的一个独特且便利的工具：它们将做出可验证主张（以及撒谎）的强大动机与一个可完全恢复的世界状态结合在一起，这种组合在开放式的语言任务中很少见。我们希望 QUACK 既能作为当前智能体的诊断工具，也能为未来的工作提供基础：例如，训练那些其讨论被明确优化以追求基于性的智能体，或者将这种验证方法扩展到更丰富的环境中。
局限性
我们的研究存在一些局限性，这些局限性也指明了未来的工作方向。主张提取依赖于一个大语言模型，且略显保守。尽管我们的人工验证表明该流程既精确（对抽样主张的提取和判定均正确）又具有高召回率（主张；§4.3），但提取器偶尔会遗漏某个主张。遗漏的主张会降低覆盖率，而不会破坏判定结果，因此我们报告的比例可能略微低估了提出的主张总数，但被评分的主张均得到了可靠的解析和判断。我们并未分离视觉模态的贡献。智能体接收的是对齐的图像和文本观测，我们也没有进行纯文本的消融实验；因此，我们将任务难度定性为长周期且部分可观测，而非将其具体归因于视觉，并将受控对比留待未来工作。我们的实验范围是有限的。我们在一个具有固定配置（, ）的单房间地图上评估了三个模型。特别是“单鸭”设置中，每局游戏内冒牌方提出的主张更少，因此“鸭”方指标所基于的样本量小于“船员”方指标。该环境支持更大的地图、更多的智能体、更多的冒牌方以及额外的角色，我们预计即使定性失败模式持续存在，绝对数值也会随这些因素而变化。最后，验证是相对于引擎的基准真相和我们的主张分类体系来定义的。无法通过任何记录事件解决的主张会被标记为“不可验证”而非进行评分，因此该框架审计的是有依据、可核查的陈述，并不试图判断自由形式对话的全部语用内容。
Ye Yuan
Rui Song
Weien Li
Zeyu Li
Haochen Liu
Xiangyu Kong
Changjiang Han
Yonghan Yang
Zichen Zhao
Zixuan Dong
Fuyuan Lyu
Bowei He
Haolun Wu
Jikun Kang
Xue Liu
McGill University,
Mila - Quebec AI Institute,
University of Cambridge,
MBZUAI - Mohamed bin Zayed University of Artificial Intelligence,
University of Toronto,
Salesforce
Corresponding to
ye.yuan3@mail.mcgill.ca
Abstract
Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent’s language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent’s ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates % of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.
QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents
Ye Yuan1, 2 ††thanks: Corresponding to ye.yuan3@mail.mcgill.ca., Rui Song1, Weien Li1, Zeyu Li1, Haochen Liu3, Xiangyu Kong1, 2, Changjiang Han4, Yonghan Yang4, Zichen Zhao4, Zixuan Dong5, Fuyuan Lyu1, 2, Bowei He4, Haolun Wu1, 2, Jikun Kang6, Xue Liu4, 1, 2 1 McGill University, 2 Mila - Quebec AI Institute, 3 University of Cambridge, 4 MBZUAI - Mohamed bin Zayed University of Artificial Intelligence, 5 University of Toronto, 6 Salesforce
1 Introduction
Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly deployed as interactive agents that must perceive their environment, communicate with other agents, decide under uncertainty, and explain their behavior in natural language (Zhu et al., 2025; Yuan et al., 2026). In such settings, an agent’s language is only useful if it stays grounded: its statements about where it has been, who it has seen, and what it has done must remain faithful to its actual perception and actions (Koh et al., 2024). This shifts the central question beyond static question answering or single-turn instruction following toward whether an agent can maintain grounding over long horizons (Curvo, 2025; Barkur et al., 2025; Jones and Bergen, 2024; Banerjee et al., 2024). In a social deduction game, players hold hidden roles and must infer the hidden roles of others from their behavior and claims. It has therefore become a natural testbed for studying reasoning, deception, coordination, and belief modeling in multi-agent settings (Hu et al., 2025; Chi et al., 2024; Fu, 2025). Compared with traditional static benchmarks, social deduction environments combine hidden information, adversarial incentives, cooperation, strategic communication, and long-horizon interaction (Yu et al., 2025; Sarkar et al., 2025). Crucially, they also admit a recoverable ground truth against which an agent’s every utterance can, in principle, be checked.
Yet existing social deduction environments for LLM agents still face two limitations that make it hard to be directly evaluated. First, most prior work evaluates agents primarily through game outcomes such as win rates, survival rates, or voting accuracy (Light et al., 2023; Wang et al., 2023). These metrics reveal little about why an agent succeeded or failed: an agent may lose despite locally coherent reasoning, or win despite producing inconsistent or unsupported claims. Second, even works that move beyond outcome-level evaluation (Song et al., 2025) remain largely text-only (Shindo et al., 2026; Xu et al., 2024a; Song et al., 2025; O’Gara, 2023). Without grounded visual observations and reconstructable trajectories, it is difficult to determine whether an agent’s dialogue is consistent with what it actually perceived and did, and thus to distinguish correct reasoning from hallucinated evidence or merely plausible dialogue patterns. As a result, important reasoning failures remain hard to identify systematically.
To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing grounded multimodal social reasoning in Vision-Language Model agents. QUACK is inspired by social deduction games such as Goose Goose Duck and recent works that leverage Among Us (Chi et al., 2024), but is purpose-built as a controlled research environment for grounded agent evaluation. Agents navigate configurable graph-based maps under partial observability, observe rendered global and local views, complete location-bound tasks, communicate through free-form discussion, and vote under hidden-role adversarial incentives. Critically, every episode is replayable through structured engine-level event logs, yielding a tick-by-tick ground-truth trajectory for each agent against which its statements can be verified.
Beyond the environment, the central contribution of QUACK is a Statement Verification Pipeline that turns this ground-truth trajectory into an automatic audit of agent language. It is embedded in a three-tier evaluation framework that measures game outcomes (Tier ), behavioral trajectories (Tier ), and utterance-level consistency (Tier ). While Tiers and provide standard outcome and behavioral context, the pipeline at Tier reconstructs each agent’s trajectory from engine logs, extracts the structured claims embedded in its discussion utterances, and verifies each claim against the reconstructed world state. This operationalizes four grounding failures as concrete, automatically measurable quantities: spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Because the audit is fully automatic, we validate it against human annotation, confirming that the reported failure rates reflect agent behavior rather than verification noise.
Using QUACK, we evaluate frontier VLM-powered agents across games in both homogeneous and cross-model adversarial settings. Our experiments show that even strong VLM agents exhibit systematic and diagnosable failures when social reasoning must remain grounded in partially observed multimodal interaction: all three frontier models hallucinate a substantial fraction of their spatial claims and make the majority of their accusations without grounded evidence.
-
We introduce QUACK, an open-source multimodal social deduction environment for auditing grounded reasoning in VLM agents, with partial observability and fully replayable logs.
-
We propose a three-tier evaluation framework scoring game outcomes, behavioral trajectories, and utterance-level consistency, moving beyond win rates toward language grounding.
-
We develop a Statement Verification Pipeline that checks each discussion utterance against the reconstructed ground-truth trajectory, operationalizing four grounding failures and validated against human annotation.
-
Across three frontier VLMs, in homogeneous and cross-model adversarial play, we show these failures arise systematically.
2 Related Work
QUACK sits at the intersection of two lines of work. We discuss social deduction games as environments for studying multi-agent language behavior and the evaluation of social agents beyond game outcomes.
Social deduction games as environments.
A large body of work uses Werewolf/Mafia-style games to study deception, persuasion, and strategic communication in LLMs, ranging from empirical studies of prompting (Xu et al., 2024a) and reasoning enhancement (Wu et al., 2024b) to dedicated evaluation arenas (Bailis et al., 2024; Shibata et al., 2023) and text-based deception games (O’Gara, 2023). A parallel line targets hidden-role deduction in Avalon, emphasizing recursive reasoning and resistance to deception (Light et al., 2023; Wang et al., 2023), while the impostor-identification setting closest to ours is explored in text-based Among Us variants (Chi et al., 2024; Fu, 2025). Beyond prompting, some works move from playing to training, using reinforcement learning to acquire strategic play and communication (Xu et al., 2024b; Sarkar et al., 2025), and others embed deduction in broader trust-and-deception or social simulations (Curvo, 2025; Park et al., 2023). Almost all of these environments, however, are text-only: agents read and write natural language. The main multimodal resource, Werewolf Among Us (Lai et al., 2023), is an observational corpus of human gameplay for modeling persuasion, rather than an interactive environment in which a vision-language agent must perceive, act, and then justify its claims. QUACK fills this gap by coupling a playable, partially observed multimodal environment with reconstructable ground-truth trajectories.
Evaluating social agents.
Most social deduction benchmarks score agents by game outcomes such as win, survival, or voting accuracy (Light et al., 2023; Wang et al., 2023; Chi et al., 2024; Fu, 2025), which reveal little about why an agent succeeds or fails. Recent work pushes beyond outcomes toward strategy quality and human alignment (Song et al., 2025), explicit opponent and belief modeling (Yu et al., 2025; Premack and Woodruff, 1978), and collaboration-competition metrics in multi-agent settings (Zhu et al., 2025; Sarkar et al., 2025), while a related thread isolates deception itself, studying lie detection (Banerjee et al., 2024) and persuasion (Jones and Bergen, 2024). Multimodal evaluation, in contrast, is largely confined to static or single-agent tasks; visual question answering (Goyal et al., 2017), chart and document understanding (Masry et al., 2022), broad multimodal benchmarks (Liu et al., 2023; Yue et al., 2024), spatial reasoning (Chen et al., 2024), and navigation or web tasks (Anderson et al., 2018; Koh et al., 2024), where there is no adversarial multi-agent dialogue to keep grounded. Methodologically, our verification procedure connects to work on faithfulness and factual consistency in text generation (Ji et al., 2023), which decomposes an output into atomic claims and checks each against an external knowledge source (Thorne et al., 2018; Min et al., 2023), or retrieves evidence to attribute and revise unsupported content (Gao et al., 2023). Unlike these settings, QUACK verifies each claim against a recoverable, agent-specific ground-truth trajectory produced by an interactive, adversarial multi-agent environment. What none of these settings provide is an utterance-level check of whether an agent’s generated claims are faithful to its own perceived-and-acted trajectory. QUACK’s Statement Verification Pipeline supplies exactly this: it reconstructs each agent’s trajectory and verifies every discussion claim against it, turning grounding failures into directly measurable quantities rather than inferring them from final outcomes.
3 The QUACK Environment
We formalize QUACK as a partially observable Markov game (Littman, 1994) played by agents on a graph-structured map. This section defines the teams and roles (§3.1), the map and state space (§3.2), the multimodal observation space (§3.3), the agent (§3.4), the action space (§3.5), and the phase-structured transition dynamics and win conditions (§3.6). We discuss the formulation here and defer the exact agent prompts to Appendix A; full configuration values are released with the code.
3.1 Agents, Teams, and Roles
A game instance has agents partitioned into two hidden-role teams, the Geese (crew) and the Ducks (impostors). At game start, of the agents are sampled uniformly at random to be Ducks and the remaining are Geese. Each agent is privately told its own role, and Ducks are additionally told the identities of their fellow Ducks, whereas Geese know only the team sizes. Our experiments use the standard configuration , , but our environment inherently allows other configurations with different values of and .
Geese.
Each Goose is assigned a private set of location-bound tasks ( in our experiments), each anchored to a specific room. A Goose wins by either collectively completing all Goose tasks or by identifying and ejecting all Ducks through discussion and voting. Geese cannot kill.
Ducks.
Ducks win when the number of living Ducks is at least the number of living Geese (voting parity). A Duck may eliminate a co-located Goose (§3.5), subject to a cooldown, and is issued a set of fake tasks identical in form to a Goose’s so that its task-like behavior is indistinguishable from a Goose’s at the level of observable actions. Ducks must blend in during free roam and avoid suspicion during meetings. The environment advances in discrete time steps, which we call ticks. We set the cooldown as ticks by default.
3.2 Map and State Space
Map.
The environment is parameterized by a map , an undirected weighted graph whose nodes are rooms and whose edges are corridors. The weight is the number of ticks required to traverse the corridor between adjacent rooms and . Figure 1 left demonstrates an omniscient view of the game state. A subset of rooms carry tasks, and one designated room holds the emergency button, which can be used to call a meeting (elaborated later in §3.5). Our environment supports configurable maps, and the instance used in our experiments is a -room map with weighted corridors with travel times ticks.
State.
The global state at tick is
| (1) |
where is the current game phase (elaborated later in §3.6), is the set of bodies currently on the map (each a tuple of victim, room, and time of death), and collects per-tick communication and witnessed-movement buffers. Each agent’s individual state records its current room, whether it is in transit along a corridor, its task progress vector, its set of visited rooms, and, for Ducks, the remaining kill cooldown. The full state is serialized to a structured engine-level event log at every tick, enabling exact replay and trajectory reconstruction.
3.3 Observation Space
QUACK is partially observable: an agent never sees the global state. At each decision point agent receives a multimodal observation consisting of two rendered images and a structured textual summary.
Rendered views.
The global map image shows the full room layout for spatial orientation but reveals no other players, only the viewer’s own position and its own task markers. The local view image renders only what the agent can presently perceive: the players and bodies in its current room, together with movement events it witnesses this tick (players departing its room or arriving into it). Figure 1 top right illustrates the local view of each agent corresponding to the left omniscient view.
Structured summary.
The text symbolically encodes the agent’s perceptual state, including information the static images cannot convey: its transit status and destination, the movement events it witnesses this tick (which players departed its room or arrived into it, and in which direction), and the adjacent rooms together with their per-corridor travel costs . It also lists the agent’s own tasks and progress, any proximity chat spoken in the room this tick, and, for Ducks, the remaining kill cooldown. Figure 1 bottom right shows an example of the structured summary from Alice’s perspective. During meetings the observation is augmented with the meeting reason, the speaking order, the discussion transcript so far, and the list of known-dead players.
3.4 Agents
Each agent is an VLM-based policy that maps observations to actions and utterances. Because the game is long-horizon and partially observed, an agent cannot rely on a single observation: at every decision point it is conditioned not only on the current observation but also on a running memory of its own trajectory so far: the sequence of rooms it has occupied, the movements it has witnessed (which players it saw depart or arrive), the players it has encountered, and the transcripts and outcomes of previous meetings. During free roam the agent receives together with this memory and selects an action (and optional utterance); during meetings it additionally conditions on the running discussion transcript before producing its statement and vote. This design means an agent’s discussion claims are generated from its own accumulated, partial recollection of the game.
3.5 Action Space
The available actions depend on the phase, the agent’s role, and its local situation. The engine exposes the legal action set with each observation.
Free-roam actions.
During free roam an agent selects one action per tick from: ; to an adjacent room , which initiates a traversal lasting ticks; , which advances the task anchored to the current room by one tick (a task completes after a fixed number of consecutive ticks in its room); , available when a body is present in the agent’s room; and , available only in the emergency-button room while a shared meeting budget remains. A Duck whose cooldown has elapsed additionally has for each co-located Goose . Orthogonally to the chosen action, an agent may attach a free-form utterance , which is heard only by agents in the same room on that tick; this is the local, "proximity chat" channel.
Meeting actions.
When a meeting is convened, free roam halts and the action space switches to language. In the discussion phase each living agent speaks in turn over a fixed number of rounds, producing a free-form natural language utterance. In the subsequent voting phase each living agent casts a vote for a player to eject or abstains.
3.6 Transition Dynamics and Win Conditions
A game proceeds as an alternation between a free-roam phase and an event-triggered meeting phase, formalized as transitions over the phase variable .
Free roam.
On each free-roam tick the engine first advances all in-transit agents (decrementing remaining travel ticks and committing arrivals), decrements Duck cooldowns, and then queries living agents in a randomized order; each chosen action is applied immediately to the state, so an agent’s action can depend on movements already resolved this tick. Movement, kills, task progress, and proximity chat all mutate the state and emit corresponding events. The phase remains FreeRoam until a body is reported or an emergency meeting is called, or until a tick budget is exhausted.
Meeting.
A or action transitions the game to Discussion: all in-transit movement is cancelled, a speaking order is fixed (the caller first, the remaining living agents shuffled), and agents speak for a fixed number of rounds. The game then enters Voting; votes are tallied and the plurality target is ejected, with ties or a plurality-abstain resulting in no ejection (Ejection). If the game is not over, surviving agents are randomly redistributed across rooms and bodies are cleared, returning the game to FreeRoam. This respawn is logged explicitly so it can be reconstructed in replay.
Win conditions.
After every phase the engine checks termination. The Ducks win immediately if living Ducks reach parity with living Geese. The Geese win if all Ducks are ejected, if all Goose tasks are completed, or if the tick budget is reached with at least one Goose alive. On termination the phase becomes GameOver and the outcome and reason are recorded.
4 Automated Evaluation Framework
A central limitation of prior social-deduction benchmarks is that they score agents almost entirely by game outcomes, which reveal little about why an agent succeeded or failed as we discussed in §2. QUACK instead evaluates agents at three complementary levels, all computed automatically from the engine-level event log of each game: Tier measures game outcomes, Tier measures behavioral trajectories, and Tier audits the groundedness of what agents say. Tiers and provide standard outcome and behavioral context; our core contribution is the Tier Statement Verification Pipeline, which reconstructs each agent’s ground-truth trajectory and checks every claim it makes during discussion against that trajectory. We summarize the metrics at each tier in Appendix B.
4.1 Tier 1: Game Outcomes
Tier records the standard outcome and summary statistics of a game directly from engine events: the winner and win condition, game length, task completion, kill and meeting counts, and survival. It also includes ejection accuracy, the fraction of ejections that removed an actual Duck, which serves as a coarse measure of collective deduction quality. These metrics situate a game but, by design, say nothing about the reasoning behind it.
4.2 Tier 2: Behavioral Trajectories
Tier reconstructs each agent’s spatial trajectory from the event log and derives behavioral statistics that outcome metrics miss. For Geese, these include voting accuracy and skip rate, task efficiency (task progress relative to the movement undertaken), spatial coverage, and the latency between a body being killed and being reported. For Ducks, they include kill rate, cooldown utilization, the rate at which a Duck reports its own victim (self-report), and post-kill displacement (the distance a Duck travels away from its kill before the next meeting). Together, Tiers and characterize what agents did; they do not test whether what agents said is consistent with it.
4.3 Tier 3: Statement Verification
The core of our framework verifies, at the level of individual utterances, whether an agent’s discussion statements are grounded in what it actually perceived and did. The pipeline has two stages: claim extraction and claim verification against the reconstructed world state.
Claim extraction.
Each free-form discussion utterance is parsed by an LLM (GPT-5.5 in our experiment) into a set of structured, individually checkable claims (Pai et al., 2024; Wu et al., 2024a). We define five claim types: (1) location: the speaker asserts that a player was in a room, or, for an ordered multi-room path, a route, (2) sighting: the speaker saw another player in a room, (3) activity: a player was doing a task, traveling, or waiting in a room, (4) accusation: the speaker suspects another player of being a Duck, and (5) defense: the speaker vouches for a player. Each claim carries a subject, the relevant room(s) or target, and a temporal reference. Extraction is run with a fixed prompt (Appendix A), de-duplicated within each utterance, and cached so that re-evaluating a game reproduces the same set.
Claim verification.
Each extracted claim is checked against the agent’s reconstructed ground-truth trajectory for the relevant time window. We recover, tick by tick, every room each agent occupied, including rooms entered only briefly while passing through, and resolve each claim’s temporal reference to a window of ticks before verifying it. Every claim receives one of five verdicts: true, false, wrong_room (the right activity in the wrong place), near_miss (a duration claim, e.g. “I was there the whole time,” that is only briefly true), or unverifiable (no ground truth resolves the claim). Location and route claims are verified by presence/ordered occupancy in the window; Sighting claims by mutual visibility; Activity claims by the logged task and movement events in the claimed room and window; and Accusation claims along two orthogonal axes (detailed in the following). Each verdict is stored with its supporting evidence, so every judgment is auditable.
| Setting | Tier 1 | Tier 2 | Tier 3: Goose (crew) | Tier 3: Duck (impostor) | |||||||||
| Goose | Duck | Goose win | Eject. acc. | Vote acc. | Cooldn. util. | Self- rep. | Goose truth. | Spat. hall. | Unsup. acc. | Lie det. | Duck truth. | Dec. rate | Dec. soph. |
| Homogeneous | |||||||||||||
| Claude-Opus-4.7 | Claude-Opus-4.7 | 90.0 | 75.0 | 76.3 | 66.7 | 8.3 | 72.3 | 10.2 | 57.8 | 46.7 | 38.8 | 11.2 | 0.0 |
| Gemini-3.1-Pro | Gemini-3.1-Pro | 66.7 | 61.7 | 67.8 | 69.8 | 1.7 | 81.6 | 15.5 | 47.9 | 83.3 | 61.8 | 27.3 | 3.1 |
| GPT-5.5 | GPT-5.5 | 76.7 | 66.7 | 69.7 | 45.0 | 2.8 | 74.9 | 12.4 | 52.6 | 73.3 | 62.4 | 20.5 | 1.7 |
| Cross-model (adversarial) | |||||||||||||
| Claude-Opus-4.7 | Gemini-3.1-Pro | 63.3 | 51.7 | 53.3 | 73.3 | 6.7 | 52.0 | 11.7 | 45.3 | 46.7 | 35.3 | 20.0 | 1.6 |
| Claude-Opus-4.7 | GPT-5.5 | 70.0 | 68.3 | 72.2 | 65.0 | 8.3 | 84.4 | 12.6 | 56.6 | 81.7 | 74.3 | 22.4 | 0.0 |
| Gemini-3.1-Pro | Claude-Opus-4.7 | 76.7 | 68.3 | 69.8 | 62.2 | 5.6 | 78.9 | 20.8 | 51.8 | 80.0 | 74.8 | 24.6 | 1.6 |
| Gemini-3.1-Pro | GPT-5.5 | 70.0 | 70.0 | 75.6 | 74.3 | 3.3 | 84.5 | 15.5 | 58.9 | 77.8 | 76.2 | 23.2 | 2.3 |
| GPT-5.5 | Claude-Opus-4.7 | 93.3 | 93.3 | 95.0 | 48.3 | 7.8 | 82.6 | 19.1 | 60.8 | 100.0 | 76.4 | 23.6 | 0.0 |
| GPT-5.5 | Gemini-3.1-Pro | 73.3 | 66.7 | 74.1 | 69.2 | 10.0 | 80.0 | 17.6 | 50.1 | 87.2 | 63.5 | 26.1 | 1.3 |
| All (270 games) | 75.6 | 69.1 | 72.7 | 63.8 | 6.0 | 76.8 | 15.1 | 53.5 | 75.2 | 62.6 | 22.1 | 1.3 | |
Validating the pipeline.
Because the audit is automatic, we assess its reliability along two axes. For precision, we draw random claims spanning all five types and have a human check, for each, both that the claim is faithfully extracted from the utterance and that its verdict is correct against the ground-truth trajectory; the pipeline is correct on of (). For recall, we draw random utterances and have a human list every claim each contains; the extractor recovers of (). Extraction is thus slightly conservative, occasionally dropping a claim, but the claims it does extract are both parsed and judged reliably, so the failure rates we report reflect agent behavior rather than verification noise.
Operationalizing grounding failures.
The verified claims let us turn four qualitative failure modes into directly measurable quantities: (1) Spatial hallucination: a Goose asserting a location or sighting that contradicts its own trajectory. (2) Unsupported accusation: accusing a player without grounded supporting evidence. We separate two axes that prior work conflates: an accusation’s outcome (did it target an actual Duck, giving accusation accuracy) and its groundedness (could the accuser actually have observed evidence against the target). (3) Deception collapse: a Duck producing easily falsifiable claims rather than subtle ones; we quantify this with the Duck deception rate and a deception sophistication score, the share of a Duck’s false claims that are near-misses rather than outright contradictions. (4) Language-action inconsistency: a stated activity or route that conflicts with the logged actions. Finally, by linking a Duck’s false claims in a meeting to the ejection that follows, we report a lie detection rate: among meetings in which a Duck told a verifiable lie, the fraction after which the Duck was ejected.
5 Experiments
We use QUACK to audit frontier VLM agents, focusing on the question our framework is built to answer: when social reasoning must stay grounded in partially observed multimodal interaction, where and how do VLM agents fail? After describing the setup (§5.1) and overall outcomes (§5.2), we organize our analysis around the four grounding failure modes operationalized in §4 (§5.3).
5.1 Experimental Setup
Models.
We evaluate three frontier vision-language models as agents: GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7. Each agent receives the multimodal observation of §3.3 and acts through the action interface of §3.5; the prompts are identical across models (Appendix A).
Settings.
We run two regimes on the -room map with agents and Duck. In the homogeneous regime all six agents are the same model ( settings). In the cross-model adversarial regime the Geese are one model and the Duck is another, over all ordered model pairs ( settings), isolating how a crew of one model fares against an impostor of another. We play games per setting, using the same set of random seeds for game initialization across settings, for games in total. Table 1 reports all settings; Table 2 aggregates the same set of metrics per model, pooling each model’s crew-side metrics over the settings in which it plays the Geese and its impostor-side metrics over the settings in which it plays the Duck ( games each). Unless noted, we report means over games, so that per-game extraction variance is averaged out at the reporting level.
5.2 Overall Outcomes
At the outcome level the games are well-balanced (Table 1). Complete results across three tiers are available in Appendix C. Across the settings, Geese win – of games and Ducks win –, so the social task is genuinely adversarial rather than trivially crew-favored. Task-related deduction is far from reliable, with ejection accuracy . As Ducks, the three models reach quite different win rates: Claude-Opus-4.7 succeeds as the impostor in only of games versus for Gemini-3.1-Pro (Table 2b).
Crucially, these outcome numbers say little on their own about the quality of an agent’s reasoning: two settings with comparable win rates can differ sharply in how grounded the underlying reasoning is. The clearest example is the strongest crew in our study, GPT-5.5, which wins of games as the Geese yet still hallucinates of its spatial claims and makes of its accusations without grounded evidence (Table 2a), failures that the win rate alone would never reveal. This is exactly the gap Tier is designed to expose.
| (a) As Goose (crew). Pooled over all settings with this model as the crew ( each). | |||||||
| Model | Goose win | Eject. acc. | Vote acc. | Goose truth. | Spatial halluc. | Unsup. accus. | Lie detect. |
| Claude-Opus-4.7 | 74.4 | 65.0 | 67.3 | 69.6 | 11.5 | 53.2 | 58.4 |
| Gemini-3.1-Pro | 71.1 | 66.7 | 71.1 | 81.7 | 17.3 | 52.9 | 80.4 |
| GPT-5.5 | 81.1 | 75.6 | 79.6 | 79.2 | 16.4 | 54.5 | 86.8 |
| (b) As Duck (impostor). Pooled over all settings with this model as the Duck ( each). | |||||||
| Model | Duck win | Cooldn. util. | Self- report | Duck truth. | Decep. rate | Decep. soph. | |
| Claude-Opus-4.7 | 13.3 | 59.1 | 7.2 | 63.3 | 19.8 | 0.5 | |
| Gemini-3.1-Pro | 32.2 | 70.8 | 6.1 | 53.5 | 24.5 | 2.0 | |
| GPT-5.5 | 27.8 | 61.4 | 4.8 | 71.0 | 22.0 | 1.3 | |
5.3 Grounding Failures (Tier 3)
Across all games, agents tell the verifiable truth most but far from all of the time: pooled Goose truthfulness is (Table 1). The interesting structure is in the failures, which fall cleanly into the four modes our pipeline operationalizes. A consistent theme is that the three frontier models share the same qualitative failure profile (Table 2), differing in degree rather than kind.
Spatial hallucination.
Even though agents are largely truthful, a substantial share of their spatial claims contradict their own trajectories: the pooled spatial hallucination rate is , i.e. roughly one in seven verifiable location/sighting claims is grounded-false. A representative example, a crew member reporting having seen a player who was already dead, is shown in Appendix D. This is the clearest evidence that the difficulty is genuinely long-horizon and partially observed: agents misremember or misreport where they were and whom they saw, the kind of error that outcome metrics cannot detect. The rate tracks model strength: as a crew, Claude-Opus-4.7 hallucinates least while GPT-5.5 and Gemini-3.1-Pro are markedly higher.
Unsupported accusation.
Accusations are both inaccurate and, more tellingly, mostly ungrounded. Appendix D gives an example where a crew member, by its own admission having seen no one, still names a suspect. In a six-player, one-Duck game, accusations land on an actual Duck less than half the time, but the sharper finding comes from separating groundedness from outcome: the pooled unsupported accusation rate is . More than half of all accusations are made without any evidence the accuser could actually have observed, regardless of whether they happen to be correct. Strikingly, this rate is remarkably stable across crews: manufacturing suspicion rather than reasoning from grounded observation is a consistent failure of all three frontier models.
Deception collapse.
On the Duck side, deception is frequent but crude. The pooled Duck deception rate is : roughly a fifth of a Duck’s verifiable claims are outright false. Appendix D shows a Duck fabricating a sighting of a player who was already dead. Critically, deception sophistication is near zero for every model in the Duck seat, meaning these lies are almost never subtle near-misses: they are flatly falsifiable against the ground truth. Ducks fabricate locations and tasks that the engine log directly contradicts, rather than constructing alibis that bend the truth. The most capable agents are thus no more sophisticated as liars. They merely lie at somewhat different rates. This "deception collapse" is precisely why a verification pipeline is informative: the lies exist and are mechanically detectable, even when the Geese fail to act on them.
Language-action inconsistency.
The same pattern appears in activity and route claims, where stated tasks and paths conflict with the logged actions. A recurring instance is a Duck claiming to have performed a task in a room where the log shows it performed none: a faked-task alibi that is internally fluent but inconsistent with what the agent actually did. Appendix D shows an example of this failure mode.
Are the lies caught?
Finally, we connect the surfaced lies back to outcomes. Among meetings in which a Duck told a verifiable lie, the Duck is subsequently ejected only (Table 1) of the time on average, and as low as (Table 2a) when Claude-Opus-4.7 is the crew. Even when a Duck’s statements are mechanically falsifiable against ground truth, Geese frequently fail to convert that into the correct ejection: a gap between the evidence available in principle and the deduction agents actually perform. Together, these results show that strong VLM agents exhibit systematic and diagnosable grounding failures that are invisible to win rates but surfaced by QUACK’s audit.
6 Conclusion and Discussion
We introduced QUACK, an open-source environment and evaluation framework for auditing whether the language of multimodal social-deduction agents stays grounded in what they actually perceived and did. Unlike prior social-deduction benchmarks, which score agents almost entirely by game outcomes, QUACK evaluates at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent’s ground-truth trajectory from engine logs and checks every discussion claim against it. This turns four qualitative failure modes: spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency into directly measurable quantities. These failures are largely shared across the three models, differing in degree rather than kind, and several of them worsen under cross-model adversarial pressure. Crucially, none of them is visible from win rates alone: two agents with similar outcomes can differ sharply in how grounded their reasoning is, and only a statement-level audit surfaces the difference.
We see two broader takeaways. First, for social-deduction and multi-agent language settings more generally, groundedness is a distinct axis of capability that is not captured by task success and deserves to be measured directly. Second, social-deduction games are a uniquely convenient instrument for studying grounded generation: they pair strong incentives to make verifiable claims (and to lie) with a fully recoverable world state, a combination rarely available in open-ended language tasks. We hope QUACK serves both as a diagnostic for current agents and as a substrate for future work: for example, training agents whose discussion is explicitly optimized for groundedness, or extending the verification approach to richer environments.
Limitations
Our study has a few limitations that also point to future work. Claim extraction relies on an LLM and is slightly conservative. Although our human validation finds the pipeline both precise (extractions and verdicts correct on sampled claims) and high-recall ( claims; §4.3), the extractor occasionally drops a claim. Missed claims reduce coverage rather than corrupt the verdicts, so our reported rates may slightly undercount the total claims made, but the claims that are scored are both parsed and judged reliably. We do not isolate the contribution of the visual modality. Agents receive aligned image and text observations, and we do not run a text-only ablation; we therefore characterize the difficulty as long-horizon and partially observed rather than attributing it specifically to vision, and leave a controlled comparison to future work. Our experimental scope is bounded. We evaluate three models on a single -room map with a fixed configuration (, ). The one-Duck setting in particular yields fewer impostor-side claims per game, so Duck metrics rest on smaller samples than crew metrics. The environment supports larger maps, more agents, more impostors, and additional roles, and we expect the absolute numbers to shift with these factors even if the qualitative failure modes persist. Finally, verification is defined relative to the engine’s ground truth and our claim taxonomy. Claims that no logged event can resolve are marked unverifiable rather than scored, so the framework audits grounded, checkable statements and does not attempt to judge the full pragmatic content of free-form dialogue.