# Whisper幻觉检测与缓解：基于隐表示引导和稀疏自编码器

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-05 08:00
- AIHOT 分数：56
- AIHOT 链接：https://aihot.virxact.com/items/cmq6l2wnb09epsl5idwl3x03n
- 原文链接：https://arxiv.org/abs/2606.07473

## AI 摘要

针对Whisper ASR模型在非语音音频上生成连贯转录（幻觉）的问题，研究提取音频编码器激活，评估原始Whisper激活和Sparse AutoEncoder（SAE）隐变量两个表示空间。两者均编码线性可分的幻觉相关信息，判别力集中在稀疏特征子集并向深层编码器递增。提出的SAE隐变量空间引导策略，在完整非语音测试集上将Whisper small幻觉率从72.63%降至14.11%，Whisper large-v3从86.88%降至27.33%，语音数据上WER退化很小，性能接近基于微调的方法。

## 正文

Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods.