LegalHalluLens:面向可信法律AI的类型化幻觉审计与校准多智能体辩论
阅读原文· arxiv.org法律AI聚合幻觉率约52%,但掩盖了错误集中方向。LegalHalluLens审计框架包含:类型化幻觉档案(数字、时间、义务权利、事实四类)、风险方向指数(RDI)及校准辩论管线。在510份合同、249,252条款实例中,同一模型内义务/数字类与时间类幻觉率差距达38-40个百分点;两个均报告52%幻觉率的系统RDI可能相反。辩论管线将虚假检测减少45%,以4B参数匹配商业API。类型档案和RDI暴露隐藏失败模式,作为多智能体辩论校准输入。
AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.