夏德国
xdg23@mails.tsinghua.edu.cn
百度
北京
李子涵
mc45085@um.edu.mo
澳门大学
澳门
赵浩辰
zhaohaochen@iie.ac.cn
中国科学院信息工程研究所
北京
谢东
xiedong04@baidu.com
百度
北京
孔宇瑶
kongyuyao@iie.ac.cn
中国科学院信息工程研究所
北京
刘希岩
liuxiyan@baidu.com
百度
北京
黄继舟
huangjizhou01@baidu.com
百度
北京
杨萌萌
yangmm˙qh@tsinghua.edu.cn
车辆与运载学院
智能绿色车辆与交通全国重点实验室
北京
杨殿阁
ydg@mail.tsinghua.edu.cn
车辆与运载学院
智能绿色车辆与交通全国重点实验室
北京
摘要。
车道级地图是自动驾驶和车道级导航的关键基础设施，然而为数百个城市构建和维护标准化车道网络仍然高度依赖人工。最近的端到端矢量化地图方法可以直接从传感器数据预测车道几何形状和拓扑结构，但它们通常将地图绘制规范和交通规则视为隐式的、依赖于数据集的监督信号。此外，在复杂场景（例如，磨损或缺失的标线以及遮挡）中，正确的车道配置往往无法仅凭视觉证据确定，这使得规范违规成为人工后期编辑的主要来源。我们提出了 MapAgent，一种工业级的智能体架构，它增强了一个矢量化主干网络，用于生成符合规范的车道地图。MapAgent 并非仅仅在地图预测上添加一个智能体循环，而是在一个受边界约束、由验证驱动的“评判者-规划者-执行者”循环中，将主干网络的感知能力与显式的规范验证、约束感知推理以及确定性的地图编辑相结合。一个视觉-语言评判者通过联合检查视觉证据和草稿向量来诊断错误，而一个工具调用的规划者则生成最小的修正编辑，并在编辑后重新验证。为了保持城市级规模生产的可扩展性，MapAgent 仅在主干网络置信度较低的图块上选择性触发，在保持吞吐量的同时增加了适度的开销。在真实世界数据集上的实验表明，与强大的生产基线相比，该方法取得了持续的改进，尤其是在复杂和长尾场景中。此外，MapAgent 已集成到百度地图中，支持全国多个城市的车道级地图生成，并将整体生产自动化率提升至超过 [原文此处数字缺失]，证明了 MapAgent 在大规模车道级地图生成方面的实用性和有效性。
地图生成；地图智能体；视觉-语言模型
第32届ACM SIGKDD知识发现与数据挖掘会议论文集 V.2；2026年8月9日至13日；韩国济州岛
第32届ACM SIGKDD知识发现与数据挖掘会议论文集 V.2 (KDD '26)，2026年8月9日至13日，韩国济州岛
应用计算 交通运输
1. 引言
车道级地图已成为自动驾驶、高级驾驶辅助以及车道级导航的核心基础设施。它们提供了关于道路几何形状、车道拓扑结构和交通控制的厘米级先验信息，使得超越感知范围的稳健规划与决策成为可能。然而，在全国范围内为数百个城市构建和维护车道级地图仍然极其昂贵。传统流程严重依赖训练有素的专家来解读制图规则和标准，并进行劳动密集型的人工标注与更新，这限制了全国范围的覆盖和更新周期（Li 等人，2022；Xia 等人，2024，2025）。近年来，端到端的高精地图学习将自动化从研究推向了工业应用：诸如 HDMapNet（Li 等人，2022）、VectorMapNet（Liu 等人，2023）、MapTR（Liao 等人，2022）和 MapTRv2（Liao 等人，2023）等方法，将多传感器输入转换为鸟瞰视角（BEV）特征，并直接解码矢量化的折线或拓扑结构，取代了大部分人工制图流程，同时实现了强劲的性能。为了满足全国范围内对规模、效率和质量的要求，在我们先前的工作中，我们开发了 DuMapNet（Xia 等人，2024），这是一个用于百度地图中城市级车道级地图生成的工业化矢量化系统（自 2023 年起部署），随后又开发了 LDMapNet-U（Xia 等人，2025），用于基于历史地图的地图更新（自 2024 年 4 月起部署）；两者共同支持超过 360 个城市，并显著降低了生产成本和更新延迟。
尽管取得了这些进展，端到端矢量化与全自动、符合规范的车道级地图生产之间仍存在关键差距。现有系统主要从监督标签中学习可见内容——车道边界、人行横道、停止线和局部拓扑结构，而工业级地图还必须满足制图标准和交通法规，要求对车道组、几何形状、属性和拓扑结构进行一致的组织。在长尾现实场景中，车道配置往往仅凭视觉证据难以确定，而需依赖基于规范和规则的先验知识，导致监督模型出现几何伪影和语义分类错误，即使在 DuMapNet 和 LDMapNet-U 等城市级部署中，仍需大量人工后期编辑。因此，商业管线仍然依赖专家编辑，他们通过交互式工具应用成文规范来修复拓扑结构并确保合规——这些能力是当前单次通过模型所不具备的，因为它们既没有显式表示此类知识，也没有提供原则性方法来推理每个场景的歧义，或决定何时以及如何调用几何编辑工具。
受这一差距驱动，我们提出从一次性端到端矢量化转向一种工业级范式，即在冻结主干模型之上进行基于智能体的精炼，其中主干模型生成草稿，而智能体层通过验证和确定性编辑来强制执行规范。在此设定下，BEV 矢量化主干模型对于可扩展性和视觉性能仍然至关重要，但它被视为草稿生成器，而非唯一负责满足制图和监管约束的组件。因此，我们通过一种可控工作流来精炼冻结主干模型的输出，该工作流结合了基于事实的诊断、规范感知验证以及基于确定性工具的编辑。为实现这一目标，我们引入了 MapAgent，这是一个在主干模型之上进行精炼的框架，通过有界迭代循环仅选择性处理困难图块。一个轻量级质量智能体利用主干模型置信度和低成本一致性检查执行早期验收；对于剩余图块，MapAgent 运行一个“评判者-规划者-执行者”循环，其中视觉语言评判者通过验证几何、拓扑和规范合规性生成结构化错误证据，规划者在能力约束下将证据转化为基于工具的编辑计划，而确定性执行者则执行编辑并重新验证，直至成功或预算耗尽。通过将端到端感知与显式验证以及严格安全约束（封闭工具、有界预算、最佳状态回退）下的规则驱动编辑相结合，MapAgent 能够生成符合领域标准的、可扩展的、生产级车道级地图。
具体而言，本文对研究和工业界做出了以下关键贡献：
-
潜在影响：我们引入 MapAgent 作为面向城市级车道级地图生成与更新的工业级智能体精炼层。MapAgent 已集成到百度地图中，支持全国多个城市的车道级地图生成与更新，并将整体生产自动化率提升至超过 。
-
新颖性：MapAgent 引入了一种工业级的智能体范式，用于车道级地图构建，专为符合规范的批量生产而设计。通过将冻结的 BEV 向量化主干与一个受限的、基于验证的循环（质量智能体 + 裁判-规划者-执行者）相结合，它对几何、拓扑和规范约束强制执行严格的合法性检查。
-
技术质量：在大型真实世界数据集上进行的大量实验表明，与强基线相比，该方法取得了持续的性能提升，尤其是在复杂和长尾场景中，同时大幅减少了人工后处理的工作量。MapAgent 在生产中的成功集成进一步验证了其在城市级部署中的鲁棒性和可扩展性。
2. MapAgent 框架
与传统的单次处理流程（BEV 观测 → 向量解码 → 人工标注 → 最终地图）相比，MapAgent 将车道级地图生成重新构想为一个可控的优化过程：骨干网络的输出被视为一个可变的地图状态，而非最终结果，如图 1 所示。该系统使用结构化的裁判来验证草稿地图，规划最小的修正动作，并通过工具执行确定性的编辑，使得整个流程高效、可解释且可投入生产。在接下来的小节中，我们首先将 MapAgent 形式化为一个受限的迭代优化过程。然后，我们介绍整体系统架构和优化工作流，随后详细描述用于约束验证的裁判智能体、用于基于工具的计划生成的规划者智能体，以及用于地图编辑的确定性执行者智能体。
2.1. 问题形式化
在车道级地图生成与更新的前期工作（Xia 等人，2024、2025）基础上，总体任务是将 BEV 观测转换为标准化的矢量车道地图。具体而言，给定从车载传感器采集的 BEV 观测数据，地图生成或更新系统旨在预测一个结构化的矢量地图，其中每个地图元素以其几何形状和属性采用统一的矢量形式表示。本文将 MapAgent 构建为一个基于冻结 BEV 矢量化主干网络的约束地图优化问题。给定 BEV 观测数据以及由主干网络生成的初始草稿地图，MapAgent 通过迭代编辑生成符合规范的地图。
在优化步骤 中，我们将 视为环境状态。设 表示由质量智能体（Quality Agent）选出的待优化的低置信度车道实例集合。
评判器（Judge）为每个 生成车道级别的结构化诊断，
| (1) |
并将其聚合为步骤级别的诊断
| (2) |
在 条件下，规划器（Planner）实现为一个基于规则的模块，生成基于工具的优化计划：
| (3) |
具体而言， 是一个有序的动作序列
| (4) |
每个动作遵循固定模式（与工人工具（Worker tools）一致），例如 ，并通过确定性执行来更新地图，即
| (5) |
所有编辑必须满足不可变的可行性门控（几何与拓扑有效性、地图规范）；导致 的编辑将被设计拒绝（详见附录 A）。总体而言，MapAgent 由经过训练的评判器（通过 SFT 和 RL 训练，见第 2.3 节）生成结构化诊断，以及基于规则的规划器在 条件下利用工人工具定义的封闭动作集生成最小有效计划来驱动。
2.2. 整体架构
如图 2 所示，MapAgent 是一个基于智能体的优化框架，构建在冻结的主干网络之上。它基于城市级 BEV 矢量化主干网络，以保持强大的感知性能和可扩展性，同时引入了一个显式的智能体验证与编辑循环，以强制执行车道级规范。质量智能体对置信度分数超过固定阈值的瓦片应用早期接受快速通道，而只有置信度低于该阈值的瓦片才会被送入有界优化循环。这种选择性路由将计算集中在困难区域，同时保持主干网络的吞吐量。在实践中，大多数瓦片被直接接受，只有少数困难瓦片需要迭代优化。快速通道接受与迭代修正之间的分离也简化了系统集成，因为主干网络保持不变。
对于每个转发的地图瓦片，MapAgent 运行一个带有重新验证的迭代式“评判者–规划者–执行者”工作流。我们明确地将外部优化预算限制为一个小的常数，在我们的系统中为三轮，以确保可预测的延迟并防止过度编辑。这种有界设计提供了稳定的运行时行为，并避免了级联修改。在每一轮中，系统在应用下一个规划之前评估当前地图状态，确保只有经过验证的编辑才会被保留。如果优化提前收敛，循环会立即终止。当预算耗尽且无法进一步获得可接受的改进时，系统会输出经过验证的最佳地图状态。
2.3. 评判者智能体：约束验证与证据生成
如图 3 所示，评判者智能体执行几何与拓扑验证，强制规范合规性，并为下游优化生成结构化证据。
对于每条被检查的车道，评判者会生成一个结构化的、标量大小的诊断条目
| (6) |
其中 是从预定义错误集 中提取的预测错误类别， 是相关的置信度， 是一个简洁的证据摘要。该分类法源自大规模的历史后期编辑日志；新观察到的异常通过人工参与流程进行审查、聚类，并用于更新地图绘制规范以及后续评判者的重训练数据。这个固定的模式是向规划者暴露的唯一接口：内部产物（例如，思维链轨迹）仅保留用于训练和调试，不会被下游模块直接使用。对所有 应用评判者会得到逐车道的诊断 ，这些诊断被聚合为 并在第 轮传递给规划者。
为了创建监督训练实例，我们从骨干草稿地图中采样单个车道，并通过将目标车道作为半透明遮罩叠加到 BEV 观测上，以聚焦于局部证据，从而构建模型输入。每个实例都配有一个真实错误标签，并可选择性地配有一个用于 SFT 的解释性参考轨迹。我们将车道的单个监督轨迹表示为 ，其中 表示第 个中间推理依据，监督语料库表示为 。请注意，车道遮罩策略是一种训练时的输入构建方法，旨在聚焦于局部证据，同时通过 中未遮罩的部分保留全局上下文。因此，Judge 是以车道为中心但并非与车道隔离的：它仍然基于完整的 BEV 观测和当前地图上下文进行条件判断，从而能够推理与相邻车道的一致性。
Judge 智能体实现为一个自回归多模态策略，它基于 BEV 、当前地图上下文 、所选车道 以及先前生成的 token 进行条件判断，以生成一个简短的推理序列，随后输出最终的结构化预测 。
我们首先通过最大化参考轨迹和最终标签的似然，对 进行监督微调。为了保持稳定性并避免对预训练骨干造成灾难性变化，在 SFT 期间仅更新轻量级适配器参数，而骨干权重保持冻结。
在 SFT 之后，我们进一步应用 GRPO 来使 Judge 与下游的 Planner–Worker 目标对齐。给定一个车道级输入 ，Judge 策略生成一个输出，该输出包含一个简洁的推理轨迹、一个最终错误类型、一个置信度分数以及一个证据摘要。对于每个输入 ，GRPO 从旧策略中采样一组候选输出，
| (7) |
并为每个候选输出 计算一个标量奖励。与 PPO 不同，GRPO 不需要单独的价值模型；相反，它通过在采样组内对奖励进行归一化来估计优势：
| (8) |
这种组内相对归一化减少了 VLM 微调期间的内存开销，并且自然适用于我们的场景，即同一车道的多个候选诊断结果可以直接进行比较。
我们优化裁剪后的替代目标
| (9) | ||||
于是 GRPO 目标函数定义为
| (10) |
其中 是冻结的 SFT 策略。KL 惩罚项防止更新后的 Judge 偏离 SFT 阶段学到的结构化输出行为。
对于每个候选输出，奖励按以下方式计算
| (11) |
我们按可执行性优先的顺序计算各项。 充当硬性门控：如果响应无法解析为 JSON，则奖励计算停止，结果为 ；如果 JSON 缺少必填字段，或证据提及的 lane 标识符不在有效元数据集内，我们也赋予一个可执行性惩罚。一个没有幻觉 lane 标识符的有效 schema 会获得一个小的格式奖励 。当预测的 error_type 与 oracle 标签匹配时 为 ，否则为 。 从零开始并检查推理链：如果推理链不包含恰好四个句子，则减去 ；如果它提及任何优先级低于最终预测的错误类型，则减去 ；如果缺少所需的高优先级排除项，则减去 。如果这些规则违反均未发生，则 。在 GRPO 训练期间，我们仅更新 LoRA 适配器参数，并保持预训练的 VLM 主干网络冻结，从而在工业地图生产约束下，在提升 Judge 对困难案例的可靠性同时保留视觉定位能力。
我们不对完整的 Judge–Planner–Worker 堆栈进行端到端训练。这是有意为之：Planner 编码了不可变更的地图规范，而 Workers 是确定性执行器，因此让它们暴露于无约束的策略优化会削弱安全保障。因此，可学习的组件仅限于 Judge，而 Planner/Worker 的行为保持可审计和版本可控。
2.4. Planner 智能体：基于工具的规划生成
Planner 智能体是一个决策模块，它将 Judge 智能体生成的车道级质量评估，在严格的地图约束下转化为可执行的细化方案。
在优化步骤中，规划器接收针对候选车道线的结构化评判输出，并利用这对信息生成修正方案。当为保持一致性而有必要时，规划器可额外查询历史地图状态，以推理过往修改。此查询起到抗振荡滤波器的作用，且仅当同一车道在连续迭代中被编辑时才会触发；在生产日志中，这种情况发生在不到优化案例的范围内。规划器无法获取除这些结构化输入之外的任何感知特征。
规划器按照固定模式输出结构化的优化方案。一个方案由公式4定义的一组有序动作序列组成，其中每个动作由工具类型、目标车道标识符以及工具特定参数定义。如果所有车道均被判定为正确，规划器将输出空方案。
所有规划决策均受外部定义且不可变的地图规则约束，这些规则编码了车道级规范和安全约束。这些规则禁止创建新车道线、跨车道组修改以及大规模结构性变更。每个生成的方案在执行前必须同时满足模式有效性和规则合规性；未通过验证的方案将被拒绝，并视为空方案处理。
规划器维护一个轻量级的内存，记录过往的优化动作及其结果，用于过滤冗余或冲突的决策。如果在既定约束和当前地图状态下无法构建有效方案，规划器将输出空方案并终止优化。这种保守的回退机制可防止级联错误，并在修正动作不确定或不可行时降低地图退化的风险。
2.5. 工作智能体：基于确定性工具的编辑
MapAgent 在有限的精化预算和封闭的、经过规范验证的工具集下运行。Worker 工具是确定性的车道线编辑器，仅支持局部的、可审计的修改。它们不能创建新的车道线实例，也不能执行跨车道组或非局部的拓扑修改。仅当车道线被诊断为冗余或虚假时，才允许删除，并且只有在通过可行性检查门限后才被接受。因此，Worker 的设计旨在避免不安全的全局拓扑变化，同时实现局部的、可审计的修正。
Worker 智能体负责执行由 Planner 智能体生成的精化计划。MapAgent 暴露了一组固定且有限的 Worker 工具，每个工具专为特定类型的车道级精化而设计：(i) 一个类别 Worker，用于语义标签修正；(ii) 一个删除 Worker，用于移除冗余或虚假的车道线；(iii) 一个平滑 Worker，用于局部几何平滑；以及 (iv) 一个再生 Worker，用于使用训练好的模型 (SAM3) (Carion et al., 2025) 进行局部几何修复。在可用的工具中，再生 Worker 是唯一利用学习模型的工具。它专为在现有车道线的空间支撑范围内进行局部几何修复而设计，其行为受 Planner 指定的约束限制，针对局部不一致性，例如断裂的线段或未对齐的几何形状。精化操作按照计划中指定的顺序依次执行。
3. 实验
在本节中，我们描述了 MapAgent 的实验设置、数据集和评估协议，随后是定量结果和分析。我们还包含了消融实验和案例研究，以更好地理解在具有挑战性的场景下的精化行为。
3.1. 实验设置
数据集构建。遵循 DuMapNet（Xia 等人，2024）中的 DuLD 数据集构建协议，我们通过以下方式从百度地图数据库构建大规模车道级矢量地图：利用相机-激光雷达融合信号的多轨迹聚合，渲染高质量离线 BEV 图像，并根据每个区域内与车道相关的实例和属性（按车道组 ID 分组，转换到局部 BEV 坐标系，并通过基本一致性操作（如移除无效几何形状和标准化属性）进行过滤）组装矢量化的真值数据。在此基础数据集之上，我们进一步使用与骨干网络无关的难度标准（例如，源自地图拓扑/场景元数据的高路口复杂度和遮挡代理指标）筛选出一个具有挑战性的困难子集，并在所有骨干网络上一致地使用该子集以确保公平性。训练集包含 3,712 张 BEV 图像，对应 59,434 个真值车道实例；DuMapNet 和 GeMap 分别生成 59,928 和 50,263 个预测车道实例。测试集包含 656 张图像，对应 10,254 个真值车道；DuMapNet 和 GeMap 分别生成 10,340 和 8,734 个预测车道实例。
评估指标。我们通过将每个预测车道匹配到最合适的真值车道，然后根据生成的匹配结果计算指标来衡量车道级修正质量。准确率、精确率、召回率和 F1 分数基于完全正确的车道定义：只有当预测车道能够匹配到真值车道且整体正确时，才将其计为真正例，即它满足匹配标准且具有正确的车道类别。没有有效匹配的预测车道被视为假正例，未匹配的真值车道被视为假负例。相比之下，BBox IoU 和 Mask IoU 用于量化几何重叠程度，Cls Acc 用于衡量类别正确性，这些指标仅针对已匹配的车道对进行计算。
骨干网络与协议。我们将 MapAgent 作为事后精炼模块，在两种具有代表性的 BEV 向量化骨干网络 GeMap 和 DuMapNet 之上进行评估。除非另有说明，骨干网络预测器保持冻结状态，MapAgent 在无需重新训练的情况下应用。对于基于 VLM 的评判智能体，我们在相同的提示词和推理协议（相同的提示词模板、解码策略和精炼预算）下，比较了不同的基础模型，包括 Qwen3-VL-Instruct (8B)、Qwen3-VL-Thinking (8B) 和 InternVL-3.5-8B。所有与精炼相关的超参数（例如，最大工具调用次数和终止条件）在实验中均保持固定。
| 评判模型 | 准确率 (%) | 精确率 / 召回率 (%) | ||||
| 无错误 | 多余车道线 | 类别错误 | 几何错误 | 结构错误 | ||
| InternVL-3.5-8B (SFT) | 58.23 | 65.00 / 82.80 | 80.00 / 49.38 | 66.67 / 54.05 | 57.14 / 40.82 | 31.87 / 55.43 |
| Qwen3-VL-8B (SFT) | 70.16 | 87.50 / 89.17 | 93.33 / 86.42 | 88.24 / 67.57 | 83.33 / 51.02 | 18.18 / 32.61 |
| Qwen3-VL-8B-Thinking (SFT) | 83.55 | 84.39 / 92.99 | 91.67 / 81.48 | 88.04 / 72.97 | 81.25 / 79.59 | 70.43 / 88.04 |
| Qwen3-VL-8B-Thinking (GRPO) | 86.01 | 92.31 / 94.90 | 96.15 / 85.80 | 93.33 / 81.08 | 87.10 / 82.65 | 66.67 / 82.61 |
| 变体 | 准确率 | 精确率 | 召回率 | F1 分数 | BBox IoU | Mask IoU | 分类准确率 |
|---|---|---|---|---|---|---|---|
| 基础预测器（无 MapAgent） | 52.5 | 71.7 | 66.5 | 68.9 | 70.4 | 35.0 | 90.0 |
| 无推理（评判模型仅预测错误类型） | 58.4 | 77.1 | 70.9 | 73.7 | 71.2 | 35.5 | 94.8 |
| 最大修正轮次 () | |||||||
| 58.3 | 76.9 | 70.8 | 73.6 | 71.2 | 35.6 | 94.5 | |
| 60.3 | 78.7 | 72.2 | 75.2 | 71.7 | 35.7 | 97.5 | |
| 62.6 | 80.7 | 73.9 | 77.0 | 71.8 | 36.0 | 98.0 | |
3.2 实现细节
硬件与软件。所有实验均在配备 NVIDIA A800 80GB GPU 的单台服务器上运行。软件栈为 PyTorch 2.6.0 与 CUDA 12.4。
VLM、SFT 与 GRPO。评判智能体模块基于 Qwen3-VL-8B-Thinking 模型（约 8B 参数）。
该模型从公开可用的预训练检查点初始化，并通过参数高效微调进一步适配。具体而言，我们首先使用 LoRA 进行监督微调（SFT），学习率为 ，批大小为 ，LoRA 秩为 ，训练轮数为 。在 SFT 检查点的基础上，我们执行一个轻量级的 GRPO 阶段，以更好地将 Judge 输出与下游地图精化目标对齐。GRPO 仅更新 LoRA 参数，学习率为 ，rollout 批大小为 ，GRPO 裁剪系数为 ，KL 系数为 ，每次更新的训练轮数为 。
运行时测量。在预热图块后，我们在随机采样的验证图块上测量运行时（包括数据 I/O）。完整的 MapAgent 流水线平均延迟为 毫秒/图块，中位数为 毫秒，p95 为 毫秒，p99 为 秒。模块级平均值为：Judge Agent 每图块 毫秒（p95 毫秒），Worker（SAM + 重新生成）每图块 毫秒（p95 毫秒）。峰值 GPU 内存使用量约为每块 A800 GB。MapAgent 在测试集中约 的图块上被触发。
3.3. 主要结果
表 1 详细展示了基于 VLM 的 Judge 在车道质量检测上的性能。SFT 在 InternVL-3.5-8B 上达到 58.23%，在 Qwen3-VL-8B 上达到 70.16%，而后续的 GRPO 对齐则带来了进一步的目标性提升。例如，GRPO 将 Qwen3-VL-8B-Thinking 的整体准确率从 83.55% 提升至 86.01%，并改善了大多数类别的精确率/召回率值。然而，结构错误类别出现了轻微下降，这表明整体决策准确率与这一少数/较难错误类型之间存在权衡。
对比结果还表明，面向推理的 VLM 更适合基于优先级的质量检测。在 SFT 下，Qwen3-VL-8B-Thinking 显著优于非思考型的 Qwen3-VL-8B，这表明显式推理有助于 Judge 区分视觉上相似的错误类型，并遵循预定义的短路顺序。经过 GRPO 后，模型在无错误、多余车道线、类别错误和几何错误上进一步提升，这表明基于奖励的对齐使 Judge 输出与可执行的下游精化更加一致。
表 3 评估了 MapAgent 作为 GeMap 和 DuMapNet 的冻结后处理优化层的效果。MapAgent 持续提升了车道线级别的正确性，且提升幅度随 Judge 能力增强而扩大。在 GeMap 上，Qwen3-VL-Thinking 将准确率从 52.8 提升至 61.3，F1 分数从 69.1 提升至 76.0。DuMapNet 的提升更为显著（准确率从 52.2 提升至 63.9，F1 分数从 68.6 提升至 78.0）。在多种模型上稳定的提升表明，MapAgent 的收益系统性地来源于更强的视觉-语言判断能力，而非特定的架构。
值得注意的是，提升主要集中在精确率、召回率和分类正确性上，而几何指标（如 IoU）保持稳定。这与 MapAgent 作为规格感知编辑器的角色相符：它修正的是虚假车道线、类别不匹配以及由局部几何/类别错误导致的匹配失败，而非激进地改变车道线几何形状。在 DuMapNet 上更显著的提升进一步表明，该优化循环在纠正初始草稿地图中的错误方面尤为有效。
3.4. 消融实验
为了更好地理解 MapAgent 中哪些部分带来了性能提升，我们在与表 3 相同的统一评估协议下进行了精简的消融实验。为便于阅读，表 2 报告了在 GeMap 和 DuMapNet 上的平均结果，并且我们仅展示后处理优化后的最终地图质量。这些消融实验聚焦于两个关键因素：Judge 是否提供超出错误标签范围的结构化证据，以及通过迭代修正能获得多大收益。
我们首先考察当评判器被简化为纯错误类型分类器时会发生什么。当我们移除显式的推理/证据生成，让评判器仅输出错误类型（无推理过程）时，系统相比冻结的基础预测器仍有显著提升，准确率从 52.5 提高到 58.4，F1 分数从 68.9 提高到 73.7，分类准确率从 90.0 上升到 94.8。然而，与完整 MapAgent 的差距依然明显：在完整的评判器–规划器–执行器循环中，F1 分数达到 77.0，分类准确率达到 98.0。这一差异表明，除了识别错误之外，评判器的结构化推理对于生成可操作、定位明确的证据至关重要，规划器才能将这些证据可靠地转化为安全、基于工具的修正。换句话说，推理不仅提升了可解释性，也提高了后续修正的可执行性。
接下来，我们在有限重试循环中改变最大修正预算。单轮修正已经能捕获大部分收益（准确率 58.3，F1 分数 73.6），表明许多错误可以通过一轮诊断和执行来修复。允许第二次迭代带来了进一步跃升（准确率 60.3，F1 分数 75.2），第三轮继续有所帮助，但收益递减（准确率 62.6，F1 分数 77.0）。这一趋势表明，迭代优化是有用的，但大多数易于修正的情况在前几轮就已解决。在这些设置下，边界框/掩码 IoU 变化很小（例如，从基线的 70.4/35.0 变为 71.8/36.0），这与 MapAgent 的保守设计一致：大部分收益来自解决离散的假阳性/假阴性和类型错误，而非激进地变形车道几何形状或改变拓扑结构。
| 地图骨干网络 | MapAgent 基座 | 准确率 | 精确率 | 召回率 | F1 分数 | 边界框 IoU | 掩码 IoU | 分类准确率 |
|---|---|---|---|---|---|---|---|---|
| GeMap | 原始预测 | 52.8 | 75.1 | 64.0 | 69.1 | 69.3 | 32.9 | 91.9 |
| InternVL-3.5-8B | 54.9 | 77.1 | 65.6 | 70.8 | 69.7 | 33.3 | 96.5 | |
| Qwen3-VL-Instruct (8B) | 56.5 | 78.6 | 66.8 | 72.2 | 69.8 | 33.5 | 96.8 | |
| Qwen3-VL-Thinking (8B) | 61.3 | 82.9 | 70.1 | 76.0 | 70.7 | 34.2 | 98.1 | |
| DuMapNet | 原始预测 | 52.2 | 68.3 | 68.9 | 68.6 | 71.4 | 37.1 | 88.0 |
| InternVL-3.5-8B | 55.0 | 70.9 | 71.1 | 71.0 | 71.9 | 37.4 | 94.6 | |
| Qwen3-VL-Instruct (8B) | 57.1 | 72.7 | 72.7 | 72.7 | 72.1 | 37.4 | 95.3 | |
| Qwen3-VL-Thinking (8B) | 63.9 | 78.4 | 77.6 | 78.0 | 72.8 | 37.7 | 97.8 |
3.5. 案例研究
图 4 展示了困难子集中的几个示例。随着场景变得更具挑战性——例如车道线模糊、缺失或被遮挡——GeMap 和 DuMapNet 的预测结果往往杂乱无章，包含虚假车道线、断裂的片段以及不一致的拓扑结构。这种失败模式并非特定于某个骨干网络，而是反映了前馈地图预测在视觉证据薄弱或模糊情况下的普遍局限性。
MapAgent 能够有效地将这些噪声预测结果优化为更清晰、结构更规整的地图。修正后的结果包含更少的虚假片段，并展现出更一致的全局拓扑结构。在实践中，优化过程侧重于明显的结构性问题——移除模型幻觉产生的车道线、抑制碎片片段并恢复连贯的布局——同时保持几何形状的保守性。这种定性趋势与车道级正确性的定量提升相符，并证明了该方法的可扩展性。
4. 讨论
MapAgent 弥合了 BEV 向量化骨干网络与符合生产规范的车道地图之间的差距。它已集成到百度地图中，用于城市级车道级地图的生成与更新，支持超过 360 个城市，并将自动化率提升至超过某个阈值。此处，自动化率定义为在固定生产周期内，完全无需人工干预即可自动完成的车道级里程数与总车道级里程数之比。与一次性预测后进行大量人工后编辑不同，MapAgent 增加了一层受约束的“判断器-规划器-执行器”优化层：在分解后的约束条件下诊断违规情况，通过一组确定的工具进行修正，并在有限的预算内重新验证。通过仅在困难图块上触发优化，它在保持吞吐量的同时，以可预测的延迟减少了人工修正的工作量。
尽管取得了这些进展，但仍存在若干值得探索的挑战，以进一步提升自主性。首先，极端的视觉模糊性可能导致车道添加和拓扑结构修改在视觉证据上欠定，这需要更强的先验知识和原则性的不确定性处理机制，以避免不安全的编辑。在当前的生产环境中，MapAgent 因此优先处理我们流程中那些可以通过确定性局部工具安全处理的主要人工后编辑类别，同时将车道添加和非局部拓扑修改留待未来工作。其次，为满足大规模工程需求，MapAgent 目前是对现有骨干网络的输出进行精炼；一个重要的方向是构建统一的智能体框架，使其能够自主调度和编排不同的专用感知骨干网络（例如，针对交叉路口、遮挡或罕见拓扑结构的专用骨干网络），并将其互补的预测结果整合到单一的约束性生成与更新流程中，从而最大化灵活性和吞吐量。
5. 相关工作
5.1. 地图构建
高清地图构建已从多阶段分割流水线演进为端到端的向量化回归（Li 等人，2022）。VectorMapNet（Liu 等人，2023）和 MapTR（Liao 等人，2022）分别利用自回归和置换不变 Transformer 架构推进了直接折线解码，而近期系统则进一步提升了几何一致性与结构化建模，例如 GeMap（Zhang 等人，2023）和 HiMap（Zhou 等人，2024）。除了纯视觉线索，越来越多的工作利用地图先验来稳定匹配并扩展范围，包括 Neural Map Prior（Xiong 等人，2023）、P-MapNet（Jiang 等人，2024）和 PriorMapNet（Wang 等人，2024），它们将 SD 地图或历史/过时先验作为额外的条件信号。最近，SDTagNet（Immel 等人，2025）利用带有文本标注的 SD 地图来增强远距离在线高清地图构建。与此同时，基于混合专家和交互的设计（例如 MapExpert（Zhang 等人，2025）和 InteractionMap（Wu 等人，2025））通过专家路由和结构化时空交互改进了长尾元素建模。同期，基于 VLM 的尝试，如 MapGPT（Zhang 等人，2024）和 MAPLM（Cao 等人，2024），表明多模态推理可能有助于理解地图和交通场景。然而，这些方法在很大程度上仍是单一的预测器，优化几何/语义指标，并将制图标准和交通规则视为隐式监督，将规范违规问题留给人工后期编辑处理。MapAgent 通过作为强 BEV 主干之上的规范感知精化层来弥补这一差距。
5.2. 智能体系统
基于大语言模型/视觉语言模型的智能体在工具使用和自我修正方面取得了快速进展。ReAct（Yao 等人，2022）将推理与行动交织进行交互式问题求解，Toolformer（Schick 等人，2023）学习决定何时以及如何调用工具，Reflexion（Shinn 等人，2023）则通过反馈驱动的记忆来改进智能体。近期工作进一步通过多轮强化学习（例如 SCoRe（Kumar 等人，2024））增强了内在的自我修正能力。另一条互补的研究路线关注模块化工具路由和基于执行的落地（例如 MRKL（Karpas 等人，2022）、SayCan（Ahn 等人，2022）、ReWOO（Xu 等人，2023）、Voyager（Wang 等人，2023））。与此同时，通用多模态基础模型（例如 GPT-4（Achiam 等人，2023）、Qwen-VL（Bai 等人，2023）、InternVL（Chen 等人，2024）、PaLM-E（Driess 等人，2023））提供了强大的视觉基础与验证能力。尽管智能体工作流已被应用于驾驶规划（例如 DriveLM（Sima 等人，2024）），但它们尚未被适配到地图生产严格的 safety 约束中。MapAgent 通过将基于 VLM 的规格验证与确定性工具执行相结合，开创了制图领域的这一范式，确保了工业级的合规性。
6. 结论
在本文中，我们提出了 MapAgent，一个工业级的智能体框架，它弥合了端到端向量化与符合规范的地图生产之间的鸿沟。受纯数据驱动主干在复杂场景中局限性的启发，我们将地图精炼形式化为一个有界、由验证驱动的迭代精炼过程，并实现了 MapAgent 作为“主干之上的精炼层”。一个轻量级的质量智能体（Quality Agent）负责选择困难图块，然后一个有界的“评判者-规划者-执行者”循环对其进行精炼：一个结构化的视觉-语言评判者（Judge）提供基于事实的诊断，一个受约束的规划者（Planner）生成基于工具的编辑方案，而确定性的执行者（Worker）则安全地执行这些方案，以确保几何/拓扑有效性和交通标准。在大型真实世界数据集上的实验表明，与强大的生产基线相比，该方法取得了持续的性能提升，尤其是在长尾场景中。此外，MapAgent 已成功集成到百度地图的生产流程中。它支持全国范围内超过 [原文此处数字缺失] 个城市的车道级地图生成与更新，将整体生产自动化率提升至超过 [原文此处数字缺失]，证明了基于智能体的精炼是大规模自动驾驶基础设施中一种可行且高效的范式。这些结果突显了在具有挑战性的条件下，结构化智能体精炼对于提升地图质量的有效性。我们相信，这一范式为未来大规模地图系统提供了一个实用且可扩展的方向。
7. 致谢
本研究得到了北京市自然科学基金（L231008, L243008）、国家自然科学基金（52472449, 52402499）、清华大学智能绿色车辆与移动国家重点实验室自主研究项目（编号：ZZ-PY-20250408）、清华大学-丰田联合中心以及清华大学-上汽通用五菱联合研究中心的资助。
参考文献
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, 等人 (2023) Gpt-4 技术报告. arXiv 预印本 arXiv:2303.08774. 引用自：§5.2
- M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman 等人 (2022) 《做我能做的，而非我说的话：将语言扎根于机器人可执行能力》。arXiv 预印本 arXiv:2204.01691。引用自：§5.2。
- J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou 和 J. Zhou (2023) 《Qwen-VL：一个用于理解、定位、文本阅读等多功能的视觉语言模型》。arXiv 预印本 arXiv:2308.12966。引用自：§5.2。
- X. Cao, T. Zhou, Y. Ma, W. Ye, C. Cui, K. Tang, Z. Cao, K. Liang, Z. Wang, J. M. Rehg 等人 (2024) 《MapLM：一个用于地图和交通场景理解的真实世界大规模视觉语言基准》。收录于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 21819–21830 页。引用自：§5.1。
- N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang 等人 (2025) 《SAM 3：基于概念分割一切》。arXiv 预印本 arXiv:2511.16719。引用自：§2.5。
- Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu 等人 (2024) 《InternVL：扩展视觉基础模型并针对通用视觉语言任务进行对齐》。收录于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 24185–24198 页。引用自：§5.2。
- D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang 等人 (2023) 《PaLM-E：一个具身多模态语言模型》。引用自：§5.2。
- F. Immel, J. Pauls, R. Fehler, F. Bieder, J. Merkert 和 C. Stiller (2025) 《SDTagNet：利用文本标注的导航地图进行在线高清地图构建》。收录于《神经信息处理系统进展》，第 38 卷。引用自：§5.1。
- Z. Jiang, Z. Zhu, P. Li, H. Gao, T. Yuan, Y. Shi, H. Zhao 和 H. Zhao (2024) 《P-MapNet：融合 SD 地图与 HD 地图先验的远见地图生成器》。IEEE 机器人与自动化快报。引用自：§5.1。
- E. Karpas, O. Abend, Y. Belinkov, B. Lenz, O. Lieber, N. Ratner, Y. Shoham, H. Bata, Y. Levine, K. Leyton-Brown 等人 (2022) MRKL 系统：一种结合大语言模型、外部知识源和离散推理的模块化神经符号架构。arXiv 预印本 arXiv:2205.00445。引用自：§5.2。
- A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs 等人 (2024) 通过强化学习训练语言模型进行自我修正。arXiv 预印本 arXiv:2409.12917。引用自：§5.2。
- Q. Li, Y. Wang, Y. Wang 和 H. Zhao (2022) Hdmapnet：一种在线高清地图构建与评估框架。载于《2022 年国际机器人与自动化大会 (ICRA)》，第 4628–4634 页。引用自：§1, §1, §5.1。
- B. Liao, S. Chen, X. Wang, T. Cheng, Q. Zhang, W. Liu 和 C. Huang (2022) Maptr：面向在线矢量化高清地图构建的结构化建模与学习。arXiv 预印本 arXiv:2208.14437。引用自：§1, §1, §5.1。
- B. Liao, S. Chen, Y. Zhang, B. Jiang, Q. Zhang, W. Liu, C. Huang 和 X. Wang (2023) Maptrv2：一种端到端的在线矢量化高清地图构建框架。arXiv 预印本 arXiv:2308.05736。引用自：§1。
- Y. Liu, T. Yuan, Y. Wang, Y. Wang 和 H. Zhao (2023) Vectormapnet：端到端矢量化高清地图学习。载于《国际机器学习大会》，第 22352–22369 页。引用自：§1, §1, §5.1。
- T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda 和 T. Scialom (2023) Toolformer：语言模型可以自学使用工具。《神经信息处理系统进展》36，第 68539–68551 页。引用自：§5.2。
- N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan 和 S. Yao (2023) Reflexion：具有语言强化学习的语言智能体。《神经信息处理系统进展》36，第 8634–8652 页。引用自：§5.2。
- C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger 和 H. Li (2024) Drivelm：基于图视觉问答的驾驶技术。载于《欧洲计算机视觉会议》，第 256–274 页。引用自：§5.2。
- G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, 和 A. Anandkumar (2023) 《Voyager：基于大语言模型的开放式具身智能体》。arXiv 预印本 arXiv:2305.16291。被 §5.2 引用。
- R. Wang, X. Lu, X. Liu, X. Zou, T. Cao, 和 Y. Li (2024) 《Priormapnet：利用先验信息增强在线矢量化高清地图构建》。arXiv 预印本 arXiv:2408.08802。被 §5.1 引用。
- K. Wu, C. Yang, 和 Z. Li (2025) 《InteractionMap：通过交互改进在线矢量化高清地图构建》。载于《计算机视觉与模式识别会议论文集》，第 17176–17186 页。被 §5.1 引用。
- D. Xia, W. Zhang, X. Liu, W. Zhang, C. Gong, J. Huang, M. Yang, 和 D. Yang (2024) 《DuMapNet：面向城市级车道级地图生成的端到端矢量化系统》。载于《第 30 届 ACM SIGKDD 知识发现与数据挖掘会议论文集》，第 6015–6024 页。被 §1、§2.1、§3.1 引用。
- D. Xia, W. Zhang, X. Liu, W. Zhang, C. Gong, X. Tan, J. Huang, M. Yang, 和 D. Yang (2025) 《LDMapNet-u：面向城市级车道级地图更新的端到端系统》。载于《第 31 届 ACM SIGKDD 知识发现与数据挖掘会议论文集 V.1》，第 2693–2702 页。被 §1、§2.1 引用。
- X. Xiong, Y. Liu, T. Yuan, Y. Wang, Y. Wang, 和 H. Zhao (2023) 《面向自动驾驶的神经地图先验》。载于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 17535–17544 页。被 §5.1 引用。
- B. Xu, Z. Peng, B. Lei, S. Mukherjee, Y. Liu, 和 D. Xu (2023) 《Rewoo：将推理与观察解耦以实现高效增强语言模型》。arXiv 预印本 arXiv:2305.18323。被 §5.2 引用。
- S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, 和 Y. Cao (2022) 《React：在语言模型中协同推理与行动》。载于《第十一届国际学习表征会议》。被 §5.2 引用。
- D. Zhang, D. Chen, P. Zhi, Y. Chen, Z. Yuan, C. Li, R. Zhou, Q. Zhou 等 (2025) 《MapExpert：利用简单高效稀疏地图元素专家进行在线高清地图构建》。载于《AAAI 人工智能会议论文集》，第 39 卷，第 14745–14753 页。被 §5.1 引用。
- Y. Zhang、Z. He、J. Li、J. Lin、Q. Guan 和 W. Yu（2024）《MapGPT：一种通过集成大语言模型与制图工具实现地图制作的自主框架》。《制图学与地理信息科学》第 51 卷第 6 期，第 717–743 页。引用于 §5.1。
- Z. Zhang、Y. Zhang、X. Ding、F. Jin 和 X. Yue（2023）《基于几何的在线矢量化高清地图构建》。arXiv 预印本 arXiv:2312.03341。引用于 §5.1。
- Y. Zhou、H. Zhang、J. Yu、Y. Yang、S. Jung、S. Park 和 B. Yoo（2024）《HiMap：面向端到端矢量化高清地图构建的混合表示学习》。载于《IEEE/CVF 计算机视觉与模式识别会议论文集》，第 15396–15406 页。引用于 §5.1。
附录 A 可行性检查与规范库
在 MapAgent 问题形式化中，我们将可行性实现为一个布尔质量门控。设 表示一个由版本化的硬性（布尔）规范谓词库 所导出的 QC 函数，该库源自行业标准与内部制图规范。我们定义
| (12) |
其中 GeoValid 和 TopoValid 是轻量级的几何/拓扑合理性检查（例如，无自相交、曲率/长度有界、车道组一致性），而每个 编码了一个不可协商的制图/交通约束。
给定一个确定性的工具转移 ，每个动作必须满足（模式/参数有效性、编辑幅度有界、车道组范围），并且只有当更新后的状态通过 QC 门控时，即 ，该状态才被接受。
附录 B 用于车道检测的 SAM3 微调
本部分描述了 SAM3 用于车道检测的微调策略。图 B.1 展示了整体架构与渐进式微调设计，其中不同阶段逐步放宽对预训练组件的优化约束。我们提供了详细的训练配置及各阶段的设计原理，随后在表 B.1 中给出了定量对比。结果表明，逐步解冻主干网络可在检测、分割和分类指标上带来持续改进。相关训练配置与示例已发布在：https://github.com/eadst/KDD-2026-MapAgent。
B.1. 训练配置
所有阶段均从同一个预训练的 SAM3 检查点（sam3.pt）初始化，并在 train.json 和 val.json 指定的车道线数据集上进行训练。我们全程启用分割监督（enable_segmentation=True），并在批处理过程中加载 RLE 掩码（with_seg_masks=True）。输入图像被调整为 1008 像素的正方形分辨率，最小缩放尺度为 480，随后填充至相同尺寸，并使用均值和标准差均设为 的参数进行归一化。训练期间，我们使用高斯噪声（标准差 0.1，上限为 20 像素）扰动输入框，并过滤掉目标为空的样本；同时将每张图像的车道线实例数量上限设为 30。
我们使用 AdamW 优化器，并启用混合精度训练（bfloat16），梯度裁剪的最大范数设为 1.0。学习率采用逆平方根调度器进行规划，包含 2000 步预热，时间尺度为 10000。Transformer 层的基础学习率设为 ，当骨干网络解冻时，对其使用更小的学习率：视觉骨干网络为 ，语言骨干网络为 。阶段 1 仅更新 Transformer 层，阶段 2 额外解冻视觉骨干网络（层间衰减系数 0.7），阶段 3 进一步解冻语言骨干网络（采用相同的保守缩放策略）。权重衰减设为 0.05，偏置项和 LayerNorm 参数不参与衰减。我们使用固定随机种子 123 训练 120 个周期。
在匹配和损失计算中，我们使用基于二元匈牙利匹配器与焦点损失，并辅以一个权重为 2.0、采用 top- 和阈值为 0.4 的辅助一对多分支。损失权重设置为： 为 3.0， 为 1.5，分类和存在性损失为 15.0，同时我们通过权重 8.0 和权重 12.0 来加强掩码监督。为了更好地拟合细长的车道线结构，掩码损失采用点采样（12544 个点，过采样率 3，重要性采样率 0.75）进行计算，将更新重点聚焦于不确定的边界区域。所有实验均使用分布式数据并行，通过 NCCL 在每节点 8 块 GPU 上进行训练，每块 GPU 的批大小为 2（全局批大小为 16），并且我们在各阶段之间保持所有其余设置不变，以确保公平比较。
B.2. 渐进式微调策略
为了使 SAM3 适应车道线检测任务，我们研究了三种微调策略，它们在允许预训练模型进行适应的程度上有所不同。我们并非将微调视为单一的配置选择，而是逐步放宽优化约束，以平衡表征稳定性与任务特定适应性。
第一阶段。我们从一种保守的设置开始，其中视觉和语言主干网络均保持固定，仅优化 Transformer 层。在此配置下，模型主要利用预训练的视觉表征，同时学习为车道线检测重组跨模态交互。这种设置强调稳定性，并提供了一个可靠的参考基准，揭示了 SAM3 在无需修改主干网络特征的情况下，对细长车道线结构进行建模的能力。
第二阶段。然后，我们允许对视觉主干网络进行有限的适应，同时保持语言主干网络冻结。具体来说，视觉主干网络以较小的学习率和逐层衰减进行训练，使得较深层的更新逐渐减弱。这种设计承认了车道线检测引入了特定领域的几何模式，包括强烈的透视效应和长程连续性，这些模式受益于对视觉特征进行适度优化，而不会破坏低层表征。
| 模型 | 准确率 (%) | 精确率 (%) | 召回率 (%) | F1 分数 (%) | 边界框 IoU (%) | 掩码 IoU (%) | 分类准确率 (%) |
|---|---|---|---|---|---|---|---|
| SAM3（阶段 1） | 50.7 | 70.2 | 60.1 | 64.8 | 67.2 | 30.8 | 92.9 |
| SAM3（阶段 2） | 54.1 | 74.5 | 63.4 | 68.5 | 68.4 | 31.9 | 94.8 |
| SAM3（阶段 3） | 58.4 | 80.1 | 67.0 | 73.0 | 69.5 | 33.0 | 96.6 |
阶段 3。最后，我们考虑一种更灵活的微调策略，其中所有主要组件都以精心缩放的学习率进行更新。Transformer 仍然是主要的优化目标，而视觉和语言主干网络的调整则更为保守。这种非对称优化反映了它们在任务中的不同角色。视觉主干网络专注于细化几何感知，而语言主干网络的有限适配则改善了文本查询与复杂场景中细粒度视觉证据之间的对齐。尽管这种策略灵活性更高，但它仍然受到良好的正则化约束，并避免与预训练语义产生过度偏离。
总体而言，这些策略构成了一个渐进式微调谱系，在稳定性和适应性之间取得了平衡。在实践中，这种设计带来了稳定的优化效果，并为理解不同模型组件对车道线检测性能的贡献提供了洞见。
附录 C 使用 GPT-5.2 生成思维链数据
为了用结构化且具有优先级意识的推理轨迹来监督裁判智能体，我们使用 GPT 5.2 作为离线数据生成器，构建了一个思维链（CoT）数据集。该模型的任务不是推断标签，而是生成简洁的视觉理由，以证明一个固定的标准结果。这种设计确保生成的 CoT 能够反映裁判的诊断逻辑，同时避免引入标签歧义或决策泄露。
C.1. 问题设定
每个训练实例对应由冻结主干模型生成的单条预测车道线。输入给 GPT 5.2 的内容包括：(i) 一张预测图像，其中包含一条高亮显示的预测车道线；(ii) 一张仅用于监督的真实图像；以及 (iii) 结构化元数据，包括预测类别、预测线周围的紧密边界框，以及指定正确性和错误类型的标准标注。
重要的是，标准标注被视为不可更改。GPT 5.2 被明确指示不得质疑、修改或推断结果，只能生成与给定标准决策一致的文本解释。
C.2. 提示词设计原则
使用单一统一的系统提示词，以确保所有生成的 CoT 样本的一致性和可控性。该提示词强制执行以下核心原则。
仅基于预测图像的视觉依据。
尽管模型可以同时使用预测图像和真实图像来理解场景，但生成的解释被限制为仅依赖预测图像中可观察到的线索。明确的禁止条款防止任何对真实值、标签或跨图像比较的引用，从而确保生成的 CoT 具有视觉依据且可安全部署。
基于优先级的短路推理。
推理过程遵循一个严格、有序的错误分类体系：
- (1)
多余车道线
- (2)
类别错误
- (3)
几何错误
- (4)
结构错误
- (5)
无错误
一旦识别出优先级更高的错误，所有优先级较低的检查都必须跳过且不得提及。这种短路结构模仿了裁判智能体在推理时的行为，并防止出现混合或内部不一致的解释。
固定长度的结构化理由。
每个 CoT 解释被限制为恰好四句话。开头的句子利用可见证据排除优先级更高的错误类型，而最后一句话则明确陈述标准错误类型作为结论。这种固定结构简化了下游解析过程，并稳定了监督训练。
具体且可验证的语言。
禁止使用抽象或模糊的表达。相反，模型被要求描述具体的、可通过图像验证的现象，例如提前终止、过度延伸、曲率偏差、间隙、错位或可见标记的部分覆盖。
C.3. 提示词实例化
在给定系统提示词的情况下，每个实例通过一个轻量级任务提示词进行实例化，该提示词注入每个样本的元数据，包括预测类别名称、边界框和标准错误类型。与标准相关的字段被标记为“不得质疑”和“不得提及”，以强化模型的作用是解释而非决策。
输出被限制为单个 JSON 对象，格式如下：
其中值包含恰好四句话。
C.4. 定性示例
图 C.1 展示了一个由 GPT 5.2 生成的代表性 CoT 示例。该解释遵循规定的优先级顺序，首先排除存在性、类别和局部几何问题，然后由于全局路径和覆盖不一致而短路于结构错误。其推理严格基于预测图像中的可观察线索，并且不引用任何真实标注。
C.5. 生成的 CoT 数据集
生成的 CoT 数据集由简洁且确定性的推理依据组成，这些依据：(i) 严格基于预测图像，(ii) 与标准错误标签完全一致，(iii) 遵循与评判智能体相同的优先级结构化决策逻辑。
在训练期间，这些 CoT 轨迹仅用作监督推理目标。在推理时，它们不会暴露给下游的计划器或工作模块，从而在可解释的诊断与可执行的系统操作之间保持清晰的分离。
附录 D 失败案例
尽管 MapAgent 在整体上取得了强劲的性能，但一些具有挑战性的案例仍然难以处理。图 D.1 展示了典型的失败案例，其中场景包含车道线证据薄弱、严重阴影、局部遮挡或高度模糊的拓扑结构。在这种情况下，主干网络的预测结果可能已经与底层车道布局存在显著偏差，使得可靠的精细化修正更加困难。由于 MapAgent 的设计目标是进行保守且基于工具的编辑，它倾向于优先保证精度和结构一致性，而非激进的拓扑重写。因此，当视觉证据极其有限或初始预测严重失真时，某些缺失、偏移或结构不一致的车道线段可能仍然无法得到解决。这些案例凸显了基于精细化修正的流程固有的局限性：最终质量仍然取决于视觉线索的可恢复性以及初始主干网络预测的质量。
附录 E 用于裁判监督的训练阶段提示词
为了训练裁判智能体生成与优先级一致的质量评估，我们使用第 C 节描述的思维链数据集对其进行监督。生成的推理过程作为参考推理轨迹，而真实错误类型和类别则提供结构化的监督目标。
训练提示词将每条预测的车道线呈现为叠加在鸟瞰图图像上的红色半透明掩膜。模型被指示将该掩膜视为一个预测假设，并基于可观察到的道路证据（包括路面标线、路缘石、沥青边界、车道连续性和局部拓扑结构）做出判断。这种设计鼓励裁判验证预测的车道线在视觉和结构上是否得到支持，而不是仅仅依赖掩膜的外观。
该提示词遵循与评判智能体相同的优先级约束评估规则。一旦确认存在更高优先级的错误，所有较低优先级的检查都会被跳过。这种短路机制可防止混合诊断，并确保训练目标与推理时的决策逻辑保持一致。对于虚拟车道线，提示词允许在即使没有可见的漆面标记的情况下，做出基于拓扑结构合理的预测，并且仅当不存在此类合理性依据时，才将其视为额外车道线。
模型需要输出一段包含在 `<think>` 标签内的推理轨迹，随后是一个最终的错误类型和一份简洁的证据摘要。推理轨迹受到生成的 CoT 推理过程的监督，而最终的错误类型和证据摘要则受到真实标注和参考解释的监督。
Deguo Xia
xdg23@mails.tsinghua.edu.cn
Baidu
Beijing
Zihan Li
mc45085@um.edu.mo
University of Macau
Macao
Haochen Zhao
zhaohaochen@iie.ac.cn
Institute of Information Engineering, Chinese Academy of Sciences
Beijing
Dong Xie
xiedong04@baidu.com
Baidu
Beijing
Yuyao Kong
kongyuyao@iie.ac.cn
Institute of Information Engineering, Chinese Academy of Sciences
Beijing
Xiyan Liu
liuxiyan@baidu.com
Baidu
Beijing
Jizhou Huang
huangjizhou01@baidu.com
Baidu
Beijing
Mengmeng Yang
yangmm˙qh@tsinghua.edu.cn
School of Vehicle and Mobility
State Key Laboratory of Intelligent Green Vehicle and Mobility
Beijing
Diange Yang
ydg@mail.tsinghua.edu.cn
School of Vehicle and Mobility
State Key Laboratory of Intelligent Green Vehicle and Mobility
Beijing
Abstract.
Lane-level maps are critical infrastructure for autonomous driving and lane-level navigation, yet constructing and maintaining standardized lane networks for hundreds of cities remains highly labor-intensive. Recent end-to-end vectorized mapping methods can predict lane geometry and topology directly from sensor data, but they typically treat mapping specifications and traffic regulations as implicit, dataset-dependent supervision. Moreover, in complex scenes (e.g., worn or missing markings and occlusions), correct lane configurations are often under-determined by visual evidence alone, making specification violations a major source of human post-editing. We propose MapAgent, an industrial-grade agentic architecture that augments a vectorization backbone for specification-compliant lane-map production. Rather than merely adding an agent loop to map prediction, MapAgent couples backbone perception with explicit specification verification, constraint-aware reasoning, and deterministic map editing under a bounded, verification-driven Judge–Planner–Worker loop. A vision–language Judge diagnoses errors by jointly inspecting visual evidence and draft vectors, while a tool-calling Planner generates minimal corrective edits with post-edit re-validation. To remain scalable for city-scale production, MapAgent is selectively triggered only on tiles with low backbone confidence, adding modest overhead while preserving throughput. Experiments on real-world datasets show consistent gains over strong production baselines, especially in complex and long-tail scenarios. Additionally, MapAgent has been integrated into Baidu Maps, supporting lane-level map generation for over cities nationwide and elevating the overall production automation to over , demonstrating MapAgent’s practicality and effectiveness for large-scale lane-level map generation.
Map Generation; Map Agent; Vision-Language Models
Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea
Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea
Applied computing Transportation
1. Introduction
Lane-level maps have become core infrastructure for autonomous driving, advanced driver assistance, and lane-level navigation. They provide centimeter-level priors on road geometry, lane topology, and traffic control, enabling robust planning and decision-making beyond the sensing horizon. However, constructing and maintaining lane-level maps for hundreds of cities at nationwide scale remains extremely expensive. Traditional pipelines rely heavily on trained experts to interpret mapping rules and standards and to conduct labor-intensive annotation and updating, which limits nationwide coverage and update cycles (Li et al., 2022; Xia et al., 2024, 2025). Recent end-to-end HD map learning has pushed automation from research to industry: methods such as HDMapNet (Li et al., 2022), VectorMapNet (Liu et al., 2023), MapTR (Liao et al., 2022) and MapTRv2 (Liao et al., 2023) convert multi-sensor inputs into bird’s-eye-view (BEV) features and directly decode vectorized polylines or topology, replacing much of the manual mapping pipeline while achieving strong performance. To meet nationwide requirements on scale, efficiency, and quality, in our prior work, we developed DuMapNet (Xia et al., 2024), an industrial vectorization system for city-scale lane-level map generation in Baidu Maps (deployed since 2023), and subsequently developed LDMapNet-U (Xia et al., 2025) for map updating conditioned on historical maps (deployed since April 2024); together, they support over 360 cities and substantially reduce production cost and update latency.
Despite these advances, a crucial gap remains between end-to-end vectorization and fully automated, specification-compliant lane-level map production. Existing systems mainly learn what is visible—lane boundaries, crosswalks, stop lines, and local topology—from supervised labels (Li et al., 2022; Liu et al., 2023; Liao et al., 2022), whereas industrial maps must also satisfy cartographic standards and traffic regulations, requiring consistent organization of lane groups, geometry, attributes, and topology. In long-tail real-world scenes (e.g., degraded/missing markings, wide unstructured pavements, adverse lighting, and occlusions), the lane configuration is frequently under-determined by visual evidence alone and instead depends on specification- and rule-based priors, leading supervised models to exhibit geometric artifacts and semantic misclassifications that still require substantial human post-editing, even in city-scale deployments such as DuMapNet and LDMapNet-U. As a result, commercial pipelines still depend on expert editors who apply codified specifications via interactive tools to repair topology and ensure compliance—capabilities that current one-pass models lack, since they neither explicitly represent such knowledge nor offer a principled way to reason about per-scene ambiguities or to decide when and how to invoke geometric editing tools.
Motivated by this gap, we propose a shift from one-pass end-to-end vectorization to an industrial-grade paradigm of agent-based refinement on top of a frozen backbone, where the backbone generates drafts and an agentic layer enforces specifications via verification and deterministic edits. In this setting, a BEV vectorization backbone remains essential for scalability and visual performance, but it is treated as a draft generator rather than the sole component responsible for satisfying cartographic and regulatory constraints. We therefore refine frozen backbone outputs with a controllable workflow that combines grounded diagnosis, specification-aware verification, and deterministic tool-based edits. To realize this, we introduce MapAgent, a refinement-on-top-of-backbone framework that selectively processes only hard tiles through a bounded iterative loop. A lightweight Quality Agent performs early acceptance using backbone confidence and cheap consistency checks; for the remaining tiles, MapAgent runs a Judge–Planner–Worker loop where a vision–language Judge produces structured error evidence by verifying geometry, topology, and specification compliance, a Planner converts evidence into a tool-grounded edit plan under capability constraints, and deterministic Workers execute edits with re-validation until success or budget exhaustion. By coupling end-to-end perception with explicit verification and rule-grounded editing under strict safety constraints (closed tools, bounded budget, best-state fallback), MapAgent enables scalable, production-quality lane-level maps aligned with domain standards.
Concretely, this paper makes the following key contributions to both the research and industrial communities:
-
Potential impact: We introduce MapAgent as an industrial-grade agentic refinement layer for city-scale lane-level map generation and updating. MapAgent has been integrated into Baidu Maps, supporting lane-level map generation and updating for over cities nationwide, and elevating the overall production automation to over .
-
Novelty: MapAgent introduces an industrial-grade agentic paradigm for lane-level mapping, designed for specification-compliant production. By coupling a frozen BEV vectorization backbone with a bounded, verification-driven loop (Quality Agent + Judge–Planner–Worker), it enforces hard validity checks for geometric, topological, and specification constraints.
-
Technical quality: Extensive experiments on large-scale real-world datasets demonstrate consistent gains over strong baselines, especially in complex and long-tail scenarios, while substantially reducing manual post-editing effort. The successful production integration of MapAgent further validates its robustness and scalability for city-scale deployment.
2. MapAgent Framework
Compared to conventional one-pass pipelines (BEV observation vector decoding human annotation final map), MapAgent reconceptualizes lane-level map generation as a controllable refinement process: the backbone output is treated as a mutable map state rather than a final result, as shown in Figure 1. The system verifies the draft map using a structured Judge, plans minimal corrective actions, and executes deterministic edits via tools, making the pipeline efficient, interpretable, and production-ready. In the following subsections, we first formalize MapAgent as a constrained iterative refinement process. We then introduce the overall system architecture and refinement workflow, followed by detailed descriptions of the Judge Agent for constraint verification, the Planner Agent for tool-based plan generation, and the deterministic Worker agents for map editing.
2.1. Problem Formulation
Following prior works on lane-level map generation and updating (Xia et al., 2024, 2025), the general task is to convert a BEV observation into a standardized vectorized lane map. Concretely, given a BEV observation collected from vehicle-mounted sensors, a map generation or updating system aims to predict a structured vectorized map , where each map element is represented in a unified vectorized form with its geometry and attributes. In this paper, we formulate MapAgent as a constrained map refinement problem built on top of a frozen BEV vectorization backbone. Given a BEV observation and an initial draft map produced by the backbone, MapAgent iteratively edits into a specification-compliant map.
At refinement step , we treat as the environment state. Let denote the set of low confidence lane instances selected by the Quality Agent for refinement.
The Judge produces a lane-wise structured diagnosis for each ,
| (1) |
and we aggregate them into a step-level diagnosis
| (2) |
Conditioned on , the Planner is implemented as a rule-based module that generates a tool-grounded refinement plan:
| (3) |
Concretely, is an ordered sequence of actions
| (4) |
Each action follows a fixed schema (consistent with Worker tools), e.g., , and the plan is executed deterministically to update the map via
| (5) |
All edits must satisfy an immutable feasibility gate (geometric and topological validity, mapping specifications); edits that lead to are rejected by design (see App. A). Overall, MapAgent is driven by a learned Judge that produces structured diagnoses (trained via SFT and RL; see Sec. 2.3), and a rule-based Planner that generates minimal valid plans under using a closed action set defined by Worker tools.
2.2. Overall Architecture
As illustrated in Figure 2, MapAgent is an agent-based refinement framework on top of a frozen backbone. It builds upon city-scale BEV vectorization backbones to retain strong perception performance and scalability, while introducing an explicit agentic verification-and-editing loop that enforces lane-level specifications. The Quality Agent applies an early-acceptance fast track to tiles whose confidence score exceeds a fixed threshold , while only tiles with confidence below are forwarded to a bounded refinement loop. This selective routing focuses computation on difficult regions while preserving the throughput of the backbone. In practice, most tiles are directly accepted, and only a minority of hard tiles require iterative refinement. The separation between fast-track acceptance and iterative correction also simplifies system integration, as the backbone remains unchanged.
For each forwarded tile, MapAgent runs an iterative Judge–Planner–Worker workflow with re-validation. We explicitly bound the outer refinement budget by a small constant, three rounds in our system, to ensure predictable latency and prevent over-editing. This bounded design provides stable runtime behavior and avoids cascading modifications. At each round, the system evaluates the current map state before applying the next plan, ensuring that only validated edits are retained. If the refinement converges earlier, the loop terminates immediately. When the budget is exhausted without further admissible improvement, the system outputs the best validated map state.
2.3. Judge Agent: Constraint Verification and Evidence Generation
As shown in Figure 3, the Judge Agent conducts geometry and topology verification, enforces specification compliance, and produces structured evidence for downstream refinement.
For each inspected lane , the Judge produces a structured, scalar-sized diagnostic entry
| (6) |
where is the predicted error category drawn from the predefined error set , is the associated confidence, and is a concise evidence summary. The taxonomy is derived from large-scale historical post-editing logs; newly observed anomalies are reviewed through a human-in-the-loop process, clustered, and used to update mapping specifications and subsequent Judge retraining data. This fixed schema is the sole interface exposed to the Planner: internal artifacts (e.g., chain-of-thought traces) are retained for training and debugging only and are not directly consumed by downstream modules. Applying the Judge to all yields lane-wise diagnoses , which are aggregated as and passed to the Planner at round .
To create supervised training instances, we sample individual lanes from the backbone draft map and construct model inputs by overlaying the target lane onto the BEV observation as a semi-transparent mask to focus attention on local evidence. Each instance is paired with a ground-truth error label and, optionally, an explanatory reference trace used for SFT. We denote a single supervised trajectory for lane as , where denotes the -th intermediate reasoning rationale, and the supervised corpus as . Note that the lane-masking strategy is a training-time input construction intended to focus attention on local evidence while preserving the global context through the unmasked portions of . Thus, the Judge is lane-centered but not lane-isolated: it still conditions on the full BEV observation and current map context, allowing it to reason about consistency with adjacent lanes.
The Judge Agent is implemented as an autoregressive multimodal policy that conditions on the BEV , the current map context , the selected lane , and previously generated tokens to produce a short reasoning sequence followed by the final structured prediction .
We first perform supervised fine-tuning on by maximizing the likelihood of the reference traces and final labels. To maintain stability and avoid catastrophic changes to the pretrained backbone, only lightweight adapter parameters are updated during SFT while the backbone weights remain frozen.
After SFT, we further apply GRPO to align the Judge with the downstream Planner–Worker objective. Given a lane-level input , the Judge policy generates an output consisting of a concise reasoning trace, a final error type, a confidence score, and an evidence summary. For each input , GRPO samples a group of candidate outputs from the old policy,
| (7) |
and computes a scalar reward for each candidate . Unlike PPO, GRPO does not require a separate value model; instead, it estimates the advantage by normalizing rewards within the sampled group:
| (8) |
This group-relative normalization reduces memory cost during VLM fine-tuning and naturally fits our setting, where multiple candidate diagnoses for the same lane can be directly compared.
We optimize the clipped surrogate
| (9) | ||||
The GRPO objective is then defined as
| (10) |
where is the frozen SFT policy. The KL penalty prevents the updated Judge from drifting away from the structured output behavior learned during SFT.
For each candidate output, the reward is computed as
| (11) |
We compute the terms in an executability-first order. acts as a hard gate: if the response cannot be parsed as JSON, the reward computation stops with ; if the JSON misses required fields or the evidence mentions lane identifiers outside the valid metadata set, we also assign a executability penalty. A valid schema with no hallucinated lane identifier receives a small format reward . is when the predicted error_type matches the oracle label and otherwise. starts from zero and checks the reasoning trace: it subtracts if the trace does not contain exactly four sentences, subtracts if it mentions any lower-priority error type than the final prediction, and subtracts if required higher-priority exclusions are missing. If none of these rule violations occurs, . During GRPO training, we update only the LoRA adapter parameters and keep the pretrained VLM backbone frozen, preserving visual grounding while improving Judge reliability on hard cases under industrial map-production constraints.
We do not train the full Judge–Planner–Worker stack end-to-end. This is intentional: the Planner encodes immutable mapping specifications and the Workers are deterministic executors, so exposing them to unconstrained policy optimization would weaken safety guarantees. The learned component is therefore restricted to the Judge, while Planner/Worker behavior remains auditable and version-controlled.
2.4. Planner Agent: Tool-Grounded Plan Generation
The Planner Agent is a decision-making module that converts lane-level quality assessments produced by the Judge Agent into executable refinement plans under strict mapping constraints.
At refinement step , the Planner receives the structured Judge outputs for candidate lane lines and consumes the pair to generate the corrective plan . When required for consistency, the Planner may additionally query the historical map state to reason about past modifications. This query acts as an anti-oscillation filter and is triggered only when the same lane is edited in consecutive iterations; in production logs, this occurs in less than of refinement cases. No perceptual features beyond these structured inputs are available to the Planner.
The Planner outputs a structured refinement plan following a fixed schema. A plan consists of an ordered sequence of actions as defined in Eq. 4, where each action is defined by a tool type, a target lane identifier, and tool-specific parameters. If all lanes are judged as correct, the Planner outputs an empty plan .
All planning decisions are constrained by externally defined and immutable mapping rules that encode lane-level specifications and safety constraints. These rules prohibit the creation of new lane lines, cross–lane-group modifications, and large-scale structural changes. Every generated plan must satisfy both schema validity and rule compliance before execution; plans that fail validation are rejected and treated as empty plans.
The Planner maintains a lightweight memory of past refinement actions and their outcomes, which is used to filter redundant or conflicting decisions. If no valid plan can be constructed under the imposed constraints and current map state, the Planner outputs an empty plan and terminates refinement. This conservative fallback mechanism prevents cascading errors and reduces the risk of degrading the map when corrective actions are uncertain or inadmissible.
2.5. Worker Agent: Deterministic Tool-Based Editing
MapAgent operates under a bounded refinement budget and a closed, specification-verified tool set. The Worker tools are deterministic lane-line editors that support only local, auditable modifications. They cannot create new lane instances or perform cross-lane-group/non-local topology modifications. Deletion is allowed only for lanes diagnosed as redundant or spurious and is accepted only after passing the feasibility gate . Therefore, the Workers are designed to avoid unsafe global topology changes while enabling local, auditable correction.
The Worker Agent is responsible for carrying out the refinement plans generated by the Planner Agent. MapAgent exposes a fixed and limited set of Worker tools, each designed for a specific type of lane-level refinement: (i) a Category Worker for semantic label correction, (ii) a Delete Worker for removing redundant or spurious lane lines, (iii) a Smooth Worker for local geometric smoothing, and (iv) a Regenerate Worker for local geometry repair using a trained model (SAM3) (Carion et al., 2025). Among the available tools, the Regenerate Worker is the only one that leverages a learned model. It is designed for local geometry repair within the spatial support of an existing lane line, and its behavior is bounded by Planner-specified constraints, targeting local inconsistencies such as broken segments or misaligned geometry. Refinement actions are executed sequentially according to the order specified in the plan .
3. Experiments
In this section, we describe the experimental setup, dataset, and evaluation protocol for MapAgent, followed by quantitative results and analysis. We also include ablations and case studies to better understand the refinement behavior under challenging scenes.
3.1. Experimental Settings
Dataset Construction. Following the DuLD dataset construction protocol in DuMapNet (Xia et al., 2024), we build large-scale lane-level vector maps from the Baidu Map Database by rendering a high-quality offline BEV image via multi-trajectory aggregation of camera–LiDAR fusion signals and assembling the vectorized ground truth from lane-related instances and attributes within each region (grouped by lane-group IDs, transformed into the local BEV coordinate system, and filtered with basic consistency operations such as removing invalid geometries and normalizing attributes). On top of this base dataset, we further curate a challenging hard subset using a backbone-agnostic difficulty criterion (e.g., high junction complexity and occlusion proxies derived from map topology / scene metadata), and use the same subset consistently across all backbones to ensure fairness. The training split contains 3,712 BEV images with 59,434 ground-truth lane instances; DuMapNet and GeMap produce 59,928 and 50,263 predicted lane instances, respectively. The test split contains 656 images with 10,254 ground-truth lanes; DuMapNet and GeMap produce 10,340 and 8,734 predicted lane instances, respectively.
Metrics. We measure lane-level correction quality by matching each predicted lane to the most appropriate ground-truth lane and then computing metrics on the resulting assignments. Accuracy, Precision, Recall, and F1 are defined on fully correct lanes: a prediction is counted as a true positive only if it can be matched to a ground-truth lane and is correct as a whole, i.e., it satisfies the matching criterion and has the correct lane category. Predictions without a valid match are treated as false positives, and unmatched ground-truth lanes are treated as false negatives. In contrast, BBox IoU and Mask IoU quantify geometric overlap, and Cls Acc measures category correctness, all computed only over matched lane pairs.
Backbones and Protocol. We evaluate MapAgent as a post-hoc refinement module on top of two representative BEV vectorization backbones, GeMap and DuMapNet. Unless otherwise noted, backbone predictors are frozen and MapAgent is applied without retraining. For the VLM-based Judge Agent, we compare different base models, including Qwen3-VL-Instruct (8B), Qwen3-VL-Thinking (8B), and InternVL-3.5-8B, under an identical prompting and inference protocol (same prompt template, decoding strategy, and refinement budget). All hyperparameters related to refinement (e.g., maximum number of tool calls and termination criteria) are fixed across experiments.
| Judge Model | Accuracy (%) | Precision / Recall (%) | ||||
| No Error | Extra Lane Line | Category Error | Geometry Error | Structure Error | ||
| InternVL-3.5-8B (SFT) | 58.23 | 65.00 / 82.80 | 80.00 / 49.38 | 66.67 / 54.05 | 57.14 / 40.82 | 31.87 / 55.43 |
| Qwen3-VL-8B (SFT) | 70.16 | 87.50 / 89.17 | 93.33 / 86.42 | 88.24 / 67.57 | 83.33 / 51.02 | 18.18 / 32.61 |
| Qwen3-VL-8B-Thinking (SFT) | 83.55 | 84.39 / 92.99 | 91.67 / 81.48 | 88.04 / 72.97 | 81.25 / 79.59 | 70.43 / 88.04 |
| Qwen3-VL-8B-Thinking (GRPO) | 86.01 | 92.31 / 94.90 | 96.15 / 85.80 | 93.33 / 81.08 | 87.10 / 82.65 | 66.67 / 82.61 |
| Variant | Accuracy | Precision | Recall | F1-score | BBox IoU | Mask IoU | Cls Acc |
|---|---|---|---|---|---|---|---|
| Base Predictor (w/o MapAgent) | 52.5 | 71.7 | 66.5 | 68.9 | 70.4 | 35.0 | 90.0 |
| w/o Reason (Judge predicts error type only) | 58.4 | 77.1 | 70.9 | 73.7 | 71.2 | 35.5 | 94.8 |
| Max Correction Rounds () | |||||||
| 58.3 | 76.9 | 70.8 | 73.6 | 71.2 | 35.6 | 94.5 | |
| 60.3 | 78.7 | 72.2 | 75.2 | 71.7 | 35.7 | 97.5 | |
| 62.6 | 80.7 | 73.9 | 77.0 | 71.8 | 36.0 | 98.0 | |
3.2. Implementation Details
Hardware and Software. All experiments were run on a single server equipped with NVIDIA A800 80GB GPUs. The software stack is PyTorch 2.6.0 with CUDA 12.4.
VLM, SFT and GRPO. The Judge Agent module is based on the model Qwen3-VL-8B-Thinking (approximately 8B parameters).
The model is initialized from a publicly available pretrained checkpoint and further adapted using parameter-efficient fine-tuning. Specifically, we first apply supervised fine-tuning (SFT) with LoRA using a learning rate of , batch size of , LoRA rank , and epochs. On top of the SFT checkpoint, we perform a lightweight GRPO stage to better align Judge outputs with downstream map-refinement objectives. GRPO updates only the LoRA parameters with learning rate , rollout batch size , GRPO clip , KL coefficient , and epochs per update.
Runtime Measurements. After warm-up tiles, we measure runtime over randomly sampled validation tiles (including data I/O). The full MapAgent pipeline achieves a mean latency of ms/tile, median ms, p95 ms, and p99 s. Module-level averages are ms per tile for the Judge Agent (p95 ms) and ms/tile for the Worker (SAM + regenerate, p95 ms). Peak GPU memory usage is approximately GB per A800. MapAgent is triggered on about of tiles in the test set.
3.3. Main Results
Table 1 details the performance of VLM-based Judges on lane-quality inspection. SFT yields 58.23% for InternVL-3.5-8B and 70.16% for Qwen3-VL-8B, while subsequent GRPO alignment yields further targeted gains. For instance, GRPO improves the overall accuracy of Qwen3-VL-8B-Thinking from 83.55% to 86.01% and improves most class-wise precision/recall values. However, the Structure Error category shows a slight drop, suggesting a trade-off between overall decision accuracy and this minority/harder error type.
The comparison also shows that reasoning-oriented VLMs are more suitable for priority-based quality inspection. Qwen3-VL-8B-Thinking substantially outperforms the non-thinking Qwen3-VL-8B under SFT, indicating that explicit reasoning helps the Judge distinguish visually similar error types and follow the predefined short-circuit order. After GRPO, the model further improves on no_error, extra_lane_line, category_error, and geometry_error, suggesting that reward-based alignment makes the Judge outputs more consistent with executable downstream refinement.
Table 3 evaluates MapAgent as a frozen post-hoc refinement layer for GeMap and DuMapNet. MapAgent consistently improves lane-level correctness, scaling with Judge capability. On GeMap, Qwen3-VL-Thinking lifts Accuracy from 52.8 to 61.3 and F1 from 69.1 to 76.0. DuMapNet sees even larger gains (Accuracy 52.263.9, F1 68.678.0). Stable improvements across various models suggest MapAgent benefits systematically from stronger visual–language judgments rather than specific architectures.
Notably, gains are concentrated in Precision, Recall, and classification correctness, while geometric metrics (e.g., IoU) remain stable. This aligns with MapAgent’s role as a specification-aware editor: it corrects spurious lanes, category mismatches, and match failures caused by local geometry/category errors rather than aggressively altering lane geometry. The pronounced improvements on DuMapNet further indicate the refinement loop is particularly effective at rectifying errors in initial draft maps.
3.4. Ablation Study
To better understand which parts of MapAgent drive the gains, we run compact ablations under the same unified evaluation protocol as Table 3. For readability, Table 2 reports the averaged results over GeMap and DuMapNet, and we only show the final map quality after post-hoc refinement. These ablations focus on two key factors: whether the Judge provides structured evidence beyond error labels, and how much benefit can be obtained from iterative correction.
We first examine what happens when the Judge is reduced to a pure error-type classifier. When we remove the explicit reasoning/evidence generation and let the Judge output the error type only (w/o Reason), the system still improves noticeably over the frozen base predictors, moving Accuracy from 52.5 to 58.4 and F1 from 68.9 to 73.7, with Cls Acc rising from 90.0 to 94.8. However, the gap to the full MapAgent remains clear: with the complete Judge–Planner–Worker loop, F1 reaches 77.0 and Cls Acc reaches 98.0. This difference suggests that, beyond recognizing what is wrong, the Judge’s structured reasoning is important for producing actionable, localized evidence that the Planner can reliably translate into safe, tool-grounded edits. In other words, reasoning improves not only interpretability, but also the executability of subsequent correction.
We then vary the maximum correction budget in the bounded retry loop. A single round already captures a large portion of the benefit (: 58.3 Accuracy and 73.6 F1), indicating many errors can be fixed with one pass of diagnosis and execution. Allowing a second iteration brings a further jump (: 60.3 Accuracy and 75.2 F1), and a third round continues to help, though with diminishing returns (: 62.6 Accuracy and 77.0 F1). This trend shows that iterative refinement is useful, but most easy-to-correct cases are resolved in the first few rounds. Across these settings, BBox/Mask IoU change only slightly (e.g., 70.4/35.0 at baseline to 71.8/36.0 at ), which aligns with MapAgent’s conservative design: most gains come from resolving discrete false positives/negatives and type mistakes rather than aggressively deforming lane geometry or altering topology.
| Map Backbone | MapAgent Base | Accuracy | Precision | Recall | F1-score | BBox IoU | Mask IoU | Cls Acc |
|---|---|---|---|---|---|---|---|---|
| GeMap | Original Prediction | 52.8 | 75.1 | 64.0 | 69.1 | 69.3 | 32.9 | 91.9 |
| InternVL-3.5-8B | 54.9 | 77.1 | 65.6 | 70.8 | 69.7 | 33.3 | 96.5 | |
| Qwen3-VL-Instruct (8B) | 56.5 | 78.6 | 66.8 | 72.2 | 69.8 | 33.5 | 96.8 | |
| Qwen3-VL-Thinking (8B) | 61.3 | 82.9 | 70.1 | 76.0 | 70.7 | 34.2 | 98.1 | |
| DuMapNet | Original Prediction | 52.2 | 68.3 | 68.9 | 68.6 | 71.4 | 37.1 | 88.0 |
| InternVL-3.5-8B | 55.0 | 70.9 | 71.1 | 71.0 | 71.9 | 37.4 | 94.6 | |
| Qwen3-VL-Instruct (8B) | 57.1 | 72.7 | 72.7 | 72.7 | 72.1 | 37.4 | 95.3 | |
| Qwen3-VL-Thinking (8B) | 63.9 | 78.4 | 77.6 | 78.0 | 72.8 | 37.7 | 97.8 |
3.5. Case Study
Figure 4 shows several examples from the hard subset. As scenes become more challenging—e.g., with faint, missing, or occluded lane markings—both GeMap and DuMapNet often produce messy predictions, including spurious lanes, fragmented segments, and inconsistent topology. This failure mode is not specific to a particular backbone, but reflects a general limitation of feed-forward map prediction under weak or ambiguous visual evidence.
MapAgent effectively refines these noisy predictions into cleaner and more structured maps. The corrected results contain fewer spurious segments and exhibit more consistent global topology. In practice, the refinement focuses on obvious structural issues—removing hallucinated lanes, suppressing fragments, and restoring coherent layouts—while keeping geometry conservative. This qualitative trend matches the quantitative gains in lane-level correctness and demonstrates the method’s scalability.
4. Discussion
MapAgent bridges the gap between BEV vectorization backbones and production-grade, specification-compliant lane maps. Integrated into Baidu Maps for city-scale lane-level map generation and updating, it supports 360+ cities and elevates automation to over . Here, the automation rate is defined as the ratio of lane-level mileage completed fully automatically without human intervention to the total lane-level mileage over a fixed production period. Instead of one-shot prediction followed by heavy human post-editing, MapAgent adds a constrained Judge–Planner–Worker refinement layer: violations are diagnosed under factorized constraints, corrected via a closed set of deterministic tools, and re-validated within a bounded budget. By triggering refinement only on hard tiles, it preserves throughput while reducing manual correction workload with predictable latency.
Despite these advances, several challenges remain worth exploring to further enhance autonomy. First, extreme visual ambiguity can make lane addition and topology modification under-determined from visual evidence, requiring stronger priors and principled uncertainty handling to avoid unsafe edits. In the current production setting, MapAgent therefore prioritizes the primary manual post-editing categories in our pipeline that can be handled safely with deterministic, local tools while deferring lane addition and non-local topology modification to future work. Second, to meet large-scale engineering requirements, MapAgent currently refines outputs from an existing backbone; an important direction is a unified agentic framework capable of autonomously scheduling and orchestrating different specialized perception backbones (e.g., specialized backbones for junctions, occlusions, or rare topologies) and integrating their complementary predictions into a single constrained generation-and-updating pipeline, thereby maximizing flexibility and throughput.
5. Related Work
5.1. Map Construction
High-definition map construction has evolved from multi-stage segmentation pipelines to end-to-end vectorized regression (Li et al., 2022). VectorMapNet (Liu et al., 2023) and MapTR (Liao et al., 2022) advanced direct polyline decoding with autoregressive and permutation-invariant transformers, respectively, while recent systems further improve geometric consistency and structured modeling, e.g., GeMap (Zhang et al., 2023) and HiMap (Zhou et al., 2024). Beyond purely visual cues, a growing line of work leverages map priors to stabilize matching and extend range, including Neural Map Prior (Xiong et al., 2023), P-MapNet (Jiang et al., 2024), and PriorMapNet (Wang et al., 2024), which incorporate SD maps or historical/outdated priors as additional conditioning signals. More recently, SDTagNet (Immel et al., 2025) exploits text-annotated SD maps to enhance far-range online HD map construction. In parallel, mixture-of-experts and interaction-based designs (e.g., MapExpert (Zhang et al., 2025) and InteractionMap (Wu et al., 2025)) improve long-tail element modeling via expert routing and structured temporal–spatial interactions. Concurrently, VLM-based efforts such as MapGPT (Zhang et al., 2024) and MAPLM (Cao et al., 2024) suggest that multimodal reasoning may help interpret maps and traffic scenes. However, these methods largely remain monolithic predictors that optimize geometric/semantic metrics and treat cartographic standards and traffic regulations as implicit supervision, leaving specification violations to manual post-editing. MapAgent addresses this gap by serving as a specification-aware refinement layer on top of strong BEV backbones.
5.2. Agentic System
LLM/VLM-based agents have rapidly advanced in tool use and self-correction. ReAct (Yao et al., 2022) interleaves reasoning and acting for interactive problem solving, Toolformer (Schick et al., 2023) learns to decide when/how to call tools, and Reflexion (Shinn et al., 2023) improves agents via feedback-driven memory. Recent work further strengthens intrinsic self-correction via multi-turn RL (e.g., SCoRe (Kumar et al., 2024)). Complementary lines study modular tool routing and grounded execution (e.g., MRKL (Karpas et al., 2022), SayCan (Ahn et al., 2022), ReWOO (Xu et al., 2023), Voyager (Wang et al., 2023)). Meanwhile, general-purpose multimodal foundations (e.g., GPT-4 (Achiam et al., 2023), Qwen-VL (Bai et al., 2023), InternVL (Chen et al., 2024), PaLM-E (Driess et al., 2023)) provide strong visual grounding and verification capabilities. While agentic workflows have been applied to driving planning (e.g., DriveLM (Sima et al., 2024)), they have not been adapted for the strict safety constraints of map production. MapAgent pioneers this paradigm in mapping by combining VLM-based specification verification with deterministic tool execution, ensuring industrial-grade compliance.
6. Conclusions
In this paper, we present MapAgent, an industrial-grade agentic framework that bridges end-to-end vectorization and specification-compliant map production. Motivated by the limitations of purely data-driven backbones in complex scenes, we formulate map refinement as a bounded, verification-driven iterative refinement process and implement MapAgent as refinement-on-top-of-backbone. A lightweight Quality Agent selects hard tiles, and a bounded Judge–Planner–Worker loop refines them: a structured vision–language Judge provides grounded diagnosis, a constrained Planner generates tool-grounded edits, and deterministic Workers execute them safely to enforce geometric/topological validity and traffic standards. Experiments on large-scale real-world datasets show consistent gains over strong production baselines, especially in long-tail scenarios. Furthermore, MapAgent has been successfully integrated into the production pipeline of Baidu Maps. Supporting lane-level map generation and updating for over cities nationwide, it has elevated the overall production automation to over , proving that agent-based refinement is a viable and efficient paradigm for large-scale autonomous driving infrastructure. These results highlight the effectiveness of structured agentic refinement for improving map quality under challenging conditions. We believe this paradigm provides a practical and scalable direction for future large-scale mapping systems.
7. Acknowledgments
This work was supported by Beijing Natural Science Foundation (L231008, L243008), National Natural Science Foundation of China (52472449, 52402499), Independent Research Project of the State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University (No. ZZ-PY-20250408), the Tsinghua University-Toyota Joint Center, and Tsinghua University–SAIC GM Wuling Joint Research Center.
References
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §5.2.
- M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. Cited by: §5.2.
- J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966. Cited by: §5.2.
- X. Cao, T. Zhou, Y. Ma, W. Ye, C. Cui, K. Tang, Z. Cao, K. Liang, Z. Wang, J. M. Rehg, et al. (2024) Maplm: a real-world large-scale vision-language benchmark for map and traffic scene understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21819–21830. Cited by: §5.1.
- N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. (2025) Sam 3: segment anything with concepts. arXiv preprint arXiv:2511.16719. Cited by: §2.5.
- Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198. Cited by: §5.2.
- D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, et al. (2023) Palm-e: an embodied multimodal language model. Cited by: §5.2.
- F. Immel, J. Pauls, R. Fehler, F. Bieder, J. Merkert, and C. Stiller (2025) SDTagNet: leveraging text-annotated navigation maps for online hd map construction. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §5.1.
- Z. Jiang, Z. Zhu, P. Li, H. Gao, T. Yuan, Y. Shi, H. Zhao, and H. Zhao (2024) P-mapnet: far-seeing map generator enhanced by both sdmap and hdmap priors. IEEE Robotics and Automation Letters. Cited by: §5.1.
- E. Karpas, O. Abend, Y. Belinkov, B. Lenz, O. Lieber, N. Ratner, Y. Shoham, H. Bata, Y. Levine, K. Leyton-Brown, et al. (2022) MRKL systems: a modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.00445. Cited by: §5.2.
- A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, et al. (2024) Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917. Cited by: §5.2.
- Q. Li, Y. Wang, Y. Wang, and H. Zhao (2022) Hdmapnet: an online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA), pp. 4628–4634. Cited by: §1, §1, §5.1.
- B. Liao, S. Chen, X. Wang, T. Cheng, Q. Zhang, W. Liu, and C. Huang (2022) Maptr: structured modeling and learning for online vectorized hd map construction. arXiv preprint arXiv:2208.14437. Cited by: §1, §1, §5.1.
- B. Liao, S. Chen, Y. Zhang, B. Jiang, Q. Zhang, W. Liu, C. Huang, and X. Wang (2023) Maptrv2: an end-to-end framework for online vectorized hd map construction. arXiv preprint arXiv:2308.05736. Cited by: §1.
- Y. Liu, T. Yuan, Y. Wang, Y. Wang, and H. Zhao (2023) Vectormapnet: end-to-end vectorized hd map learning. In International Conference on Machine Learning, pp. 22352–22369. Cited by: §1, §1, §5.1.
- T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36, pp. 68539–68551. Cited by: §5.2.
- N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36, pp. 8634–8652. Cited by: §5.2.
- C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li (2024) Drivelm: driving with graph visual question answering. In European conference on computer vision, pp. 256–274. Cited by: §5.2.
- G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Cited by: §5.2.
- R. Wang, X. Lu, X. Liu, X. Zou, T. Cao, and Y. Li (2024) Priormapnet: enhancing online vectorized hd map construction with priors. arXiv preprint arXiv:2408.08802. Cited by: §5.1.
- K. Wu, C. Yang, and Z. Li (2025) InteractionMap: improving online vectorized hdmap construction with interaction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17176–17186. Cited by: §5.1.
- D. Xia, W. Zhang, X. Liu, W. Zhang, C. Gong, J. Huang, M. Yang, and D. Yang (2024) DuMapNet: an end-to-end vectorization system for city-scale lane-level map generation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6015–6024. Cited by: §1, §2.1, §3.1.
- D. Xia, W. Zhang, X. Liu, W. Zhang, C. Gong, X. Tan, J. Huang, M. Yang, and D. Yang (2025) LDMapNet-u: an end-to-end system for city-scale lane-level map updating. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 2693–2702. Cited by: §1, §2.1.
- X. Xiong, Y. Liu, T. Yuan, Y. Wang, Y. Wang, and H. Zhao (2023) Neural map prior for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17535–17544. Cited by: §5.1.
- B. Xu, Z. Peng, B. Lei, S. Mukherjee, Y. Liu, and D. Xu (2023) Rewoo: decoupling reasoning from observations for efficient augmented language models. arXiv preprint arXiv:2305.18323. Cited by: §5.2.
- S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §5.2.
- D. Zhang, D. Chen, P. Zhi, Y. Chen, Z. Yuan, C. Li, R. Zhou, Q. Zhou, et al. (2025) Mapexpert: online hd map construction with simple and efficient sparse map element expert. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 14745–14753. Cited by: §5.1.
- Y. Zhang, Z. He, J. Li, J. Lin, Q. Guan, and W. Yu (2024) MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools. Cartography and Geographic Information Science 51 (6), pp. 717–743. Cited by: §5.1.
- Z. Zhang, Y. Zhang, X. Ding, F. Jin, and X. Yue (2023) Online vectorized hd map construction using geometry. arXiv preprint arXiv:2312.03341. Cited by: §5.1.
- Y. Zhou, H. Zhang, J. Yu, Y. Yang, S. Jung, S. Park, and B. Yoo (2024) Himap: hybrid representation learning for end-to-end vectorized hd map construction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15396–15406. Cited by: §5.1.
Appendix A Feasibility Check and Specification Library
In MapAgent problem formulation, we implement feasibility as a boolean quality gate. Let denote a QC function induced by a versioned library of hard (boolean) specification predicates , derived from industry standards and internal cartographic specifications. We define
| (12) |
where GeoValid and TopoValid are lightweight geometric / topological sanity checks (e.g., no self-intersection, bounded curvature/length, lane-group consistency), and each encodes a non-negotiable cartographic / traffic constraint.
Given a deterministic tool transition , each action must satisfy (schema / parameter validity, bounded edit magnitude, lane-group scope), and an updated state is accepted only if it passes the QC gate, i.e., .
Appendix B SAM3 Fine-tuning for Lane Detection
This part describes the fine-tuning strategy of SAM3 for lane detection. Figure B.1 illustrates the overall architecture and the progressive fine-tuning design, where different stages gradually relax the optimization constraints on the pretrained components. We provide detailed training configurations and the rationale behind each stage, followed by a quantitative comparison in Table B.1. The results demonstrate that progressively unfreezing the backbones leads to consistent improvements across detection, segmentation, and classification metrics. Related training configurations and examples have been released at: https://github.com/eadst/KDD-2026-MapAgent.
B.1. Training setup
All stages are initialized from the same pretrained SAM3 checkpoint (sam3.pt) and trained on the lane dataset specified by train.json and val.json. We enable segmentation supervision throughout (enable_segmentation=True) and load RLE masks during batching (with_seg_masks=True). Input images are resized to a square resolution of 1008 with a minimum scale of 480, followed by padding to the same size and normalization with mean and std both set to . During training, we perturb input boxes with Gaussian noise (std 0.1, capped at 20 pixels) and filter samples with empty targets; we also cap the maximum number of lane instances per image to 30.
We optimize with AdamW and mixed precision enabled (bfloat16), using gradient clipping with max norm 1.0. The learning rate is scheduled by an inverse square root scheduler with 2000 warmup steps and timescale 10000. We use a base learning rate of for transformer layers, and apply smaller learning rates for backbones when they are unfrozen: for the vision backbone and for the language backbone. Stage 1 updates only the transformer layers, Stage 2 additionally unfreezes the vision backbone with layer-wise decay 0.7, and Stage 3 further unfreezes the language backbone with the same conservative scaling. Weight decay is set to 0.05, while biases and LayerNorm parameters are excluded from decay. We train for 120 epochs with a fixed random seed of 123.
For matching and loss computation, we use a binary Hungarian matcher with focal loss , augmented by an auxiliary one-to-many branch with weight 2.0, top-, and threshold 0.4. The loss weights are set to 3.0 for and 1.5 for , 15.0 for classification and presence losses, and we strengthen mask supervision with weight 8.0 and weight 12.0. To better fit thin lane structures, mask loss is computed with point sampling (12544 points, oversample ratio 3, importance sampling ratio 0.75), focusing updates on uncertain boundary regions. All experiments are trained with distributed data parallel using NCCL on 8 GPUs per node, batch size 2 per GPU (global batch 16), and we keep all remaining settings fixed across stages to ensure fair comparison.
B.2. Progressive Fine-tuning Strategies
To adapt SAM3 to the lane detection task, we investigate three fine-tuning strategies that differ in how much of the pretrained model is allowed to adapt. Rather than treating fine-tuning as a single configuration choice, we gradually relax the optimization constraints to balance representation stability and task-specific adaptability.
Stage 1. We begin with a conservative setting in which both the vision and language backbones are kept fixed, and only the transformer layers are optimized. In this configuration, the model primarily leverages pretrained visual representations while learning to reorganize cross-modal interactions for lane detection. This setting emphasizes stability and provides a reliable reference, revealing the extent to which SAM3 can model thin and elongated lane structures without modifying its backbone features.
Stage 2. We then allow limited adaptation of the vision backbone while keeping the language backbone frozen. Specifically, the vision backbone is trained with a smaller learning rate and layer-wise decay, such that deeper layers receive progressively weaker updates. This design acknowledges that lane detection introduces domain-specific geometric patterns, including strong perspective effects and long-range continuity, which benefit from moderate refinement of visual features without disrupting low-level representations.
| Model | Accuracy (%) | Precision (%) | Recall (%) | F1-score (%) | BBox IoU (%) | Mask IoU (%) | Cls Acc (%) |
|---|---|---|---|---|---|---|---|
| SAM3 (Stage 1) | 50.7 | 70.2 | 60.1 | 64.8 | 67.2 | 30.8 | 92.9 |
| SAM3 (Stage 2) | 54.1 | 74.5 | 63.4 | 68.5 | 68.4 | 31.9 | 94.8 |
| SAM3 (Stage 3) | 58.4 | 80.1 | 67.0 | 73.0 | 69.5 | 33.0 | 96.6 |
Stage 3. Finally, we consider a more flexible fine-tuning strategy in which all major components are updated with carefully scaled learning rates. The transformer remains the primary optimization target, while the vision and language backbones are adjusted more conservatively. This asymmetric optimization reflects their different roles in the task. The vision backbone focuses on refining geometric perception, whereas limited adaptation of the language backbone improves alignment between textual queries and fine-grained visual evidence in complex scenes. Despite its increased flexibility, this strategy remains well regularized and avoids excessive drift from pretrained semantics.
Overall, these strategies form a progressive fine-tuning spectrum that balances stability and adaptability. In practice, this design leads to stable optimization and provides insights into the contribution of different model components to lane detection performance.
Appendix C Chain-of-Thought Data Generation with GPT-5.2
To supervise the Judge Agent with structured and priority-aware reasoning traces, we construct a chain-of-thought (CoT) dataset using GPT 5.2 as an offline data generator. Rather than inferring labels, the model is tasked with producing concise visual rationales that justify a fixed oracle outcome. This design ensures that the generated CoT reflects the Judge’s diagnostic logic without introducing label ambiguity or decision leakage.
C.1. Problem Setup
Each training instance corresponds to a single predicted lane line produced by the frozen backbone model. The input to GPT 5.2 includes: (i) a Pred image containing exactly one highlighted predicted lane line, (ii) a GT image used for supervision only, and (iii) structured metadata, including the predicted category, a tight bounding box around the predicted line, and an oracle annotation specifying correctness and error type.
Importantly, the oracle annotation is treated as immutable. GPT 5.2 is explicitly instructed not to question, revise, or infer the outcome, but only to generate a textual explanation that is consistent with the given oracle decision.
C.2. Prompt Design Principles
A single unified system prompt is used to ensure consistency and controllability across all generated CoT samples. The prompt enforces the following core principles.
Pred-only visual grounding.
Although both Pred and GT images are available to the model to understand the scene, the generated explanation is constrained to rely only on observable cues in the Pred image. Explicit prohibitions prevent any reference to ground truth, labels, or cross-image comparison, ensuring that the resulting CoT is visually grounded and safe for deployment.
Priority-based short-circuit reasoning.
The reasoning process follows a strict, ordered error taxonomy:
- (1)
extra_lane_line
- (2)
category_error
- (3)
geometry_error
- (4)
structure_error
- (5)
no_error
Once a higher-priority error is identified, all lower-priority checks must be skipped and must not be mentioned. This short-circuit structure mirrors the Judge Agent’s inference-time behavior and prevents mixed or internally inconsistent explanations.
Fixed-length structured rationale.
Each CoT explanation is constrained to exactly four sentences. The initial sentences rule out higher-priority error types using visible evidence, while the final sentence explicitly states the oracle error type as a conclusion. This fixed structure simplifies downstream parsing and stabilizes supervised training.
Concrete and verifiable language.
Abstract or hedging expressions are disallowed. Instead, the model is required to describe concrete, image-verifiable phenomena such as early termination, over-extension, curvature deviation, gaps, misalignment, or partial coverage of visible markings.
C.3. Prompt Instantiation
Given the system prompt, each instance is instantiated with a lightweight task prompt that injects per-sample metadata, including the predicted category name, bounding box, and oracle error type. Oracle-related fields are marked as do not question and do not mention, reinforcing that the model’s role is explanation rather than decision-making.
The output is constrained to a single JSON object of the form:
where the value contains exactly four sentences.
C.4. Qualitative Example
Figure C.1 illustrates a representative CoT example generated by GPT 5.2. The explanation follows the prescribed priority order, first ruling out existence, category, and local geometry issues, and then short-circuiting at a structure_error due to global path and coverage inconsistency. The rationale is strictly grounded in observable cues from the Pred image and does not reference any ground-truth annotations.
C.5. Resulting CoT Dataset
The resulting CoT dataset consists of concise and deterministic rationales that: (i) are strictly grounded in the Pred image, (ii) align exactly with the oracle error labels, and (iii) follow the same priority-structured decision logic as the Judge Agent.
During training, these CoT traces are used only as supervised reasoning targets. They are not exposed to downstream Planner or Worker modules at inference time, preserving a clean separation between interpretable diagnosis and executable system actions.
Appendix D Bad Cases
Although MapAgent achieves strong overall performance, some challenging cases remain difficult. Figure D.1 shows representative failure cases where the scene contains weak lane evidence, severe shadows, local occlusions, or highly ambiguous topology. In such situations, the backbone predictions may already deviate substantially from the underlying lane layout, making reliable refinement more difficult. Since MapAgent is designed to apply conservative and tool-grounded edits, it tends to prioritize precision and structural consistency over aggressive topology rewriting. As a result, when the visual evidence is extremely limited or the initial prediction is severely corrupted, certain missing, shifted, or structurally inconsistent lane segments may still remain unresolved. These cases highlight an inherent limitation of refinement-based pipelines: the final quality still depends on both the recoverability of visual cues and the quality of the initial backbone prediction.
Appendix E Training-Time Prompt for Judge Supervision
To train the Judge Agent to produce priority-consistent quality assessments, we supervise it with the CoT dataset described in Section C. The generated rationales are used as reference reasoning traces, while the oracle error type and category provide structured supervision targets.
The training prompt presents each predicted lane line as a red semi-transparent mask overlaid on the BEV image. The model is instructed to treat the mask as a prediction hypothesis and to base its judgment on observable road evidence, including painted markings, curbs, asphalt boundaries, lane continuity, and local topology. This design encourages the Judge to verify whether the predicted lane is visually and structurally supported, rather than relying on the mask appearance alone.
The prompt follows the same priority-constrained evaluation rule used by the Judge Agent. Once a higher-priority error is confirmed, all lower-priority checks are skipped. This short-circuit mechanism prevents mixed diagnoses and keeps the training target consistent with inference-time decision logic. For virtual lines, the prompt allows predictions that are topologically justified even without visible painted markings, and treats them as extra_lane_line only when no such justification exists.
The model is required to output a reasoning trace enclosed within <think> tags, followed by one final error type and a concise evidence summary. The reasoning trace is supervised by the generated CoT rationales, whereas the final error type and evidence summary are supervised by oracle annotations and reference explanations.