用马氏余弦相似度比较线性探针
阅读原文· arxiv.org线性探针常通过余弦相似度比较,马氏余弦相似度(MCS)利用测试数据协方差重新加权内积,是一种任务感知改进。Ying等人(2026)发现探针的MCS与分布外(OOD)参考探针MCS近乎完美线性预测OOD AUROC(R²=0.98)。本文将这一发现扩展到不同模型、层和概念域,并证明在投影为高斯分布的平衡类中,OOD AUROC与参考探针MCS呈线性关系,两者均为探针在测试数据上信噪比的sigmoid函数。理论还预测并实验验证了线性失效的条件。MCS为比较线性探针提供了兼具理论和实证效果的替代方案。
Linear probes are widely used in interpretability research and often compared by cosine similarity. The Mahalanobis cosine similarity (MCS) between two directions, which reweights the inner product by test data covariance, is a natural task-aware refinement. Ying et al. (2026) report that a probe's MCS to a reference probe trained on the out-of-distribution (OOD) data near-perfectly linearly predicts the probe's OOD AUROC (R^2 = 0.98). Here, we extend this empirical finding across models, layers, and concept domains, and prove this general phenomenon in closed form: For balanced classes whose projections are Gaussian, OOD AUROC and MCS to the reference probe are linear because both are sigmoid-shaped functions of the probe's signal-to-noise ratio (SNR) on the test data. The theory also predicts when this linearity fails, which we verify empirically. MCS offers a theoretically grounded and empirically effective alternative to Euclidean cosine similarity for comparing linear probes.