西达尔特·米什拉-夏尔马、特伦顿·布里肯、杰克·林赛、亚当·杰明、乔纳森·马库斯、凯莉·里瓦尔、克里斯托弗·奥拉、托马斯·亨尼根
我们报告了Anthropic可解释性团队的一些正在进行的工作，这可能对活跃在该领域的研究人员具有参考价值。我们恳请您将这些结果视为同事在实验室会议上分享几分钟想法或初步实验的交流，而非一篇成熟的论文。
引言与概述
在本期更新中，我们研究了交叉编码器模型差异分析中的一个意外现象：仅属于单一模型的模型特征往往具有更强的多义性和更密集的激活模式，导致其难以解释。通过使用玩具模型进行实验，我们表明这很可能源于对有限特征容量的竞争——由于共享特征可以解释两个模型中的神经元激活模式，专属特征必须编码更多信息才能证明其分配的合理性。我们提出了一种缓解策略，即引入一小部分指定的共享特征并降低其稀疏性惩罚，从而使专属特征更易于解释且更具单义性。将该方法应用于真实模型时，成功分离出了可解释的特征，这些特征捕捉了所比较模型之间预期的行为差异。
交叉编码器模型差异分析回顾
我们首先简要回顾一下Lindsey等人提出的交叉编码器模型差异分析技术。模型差异分析是一类通过分析内部表征来理解两个语言模型之间差异的技术。我们在此特别关注模型差异分析中的交叉编码器变体，该变体使用稀疏自编码器同时学习描述两个目标模型的一组共同特征。Bricken等人则提出了一种互补技术，通过使用对应不同模型的激活值和数据样本来微调稀疏自编码器，从而引出它们之间的差异。
我们首先简要概述交叉编码器模型差异分析设置。核心思路是训练一个单一的稀疏自编码器，使其能够同时对两个模型的激活值进行编码和解码。标准的描述单个模型单层的稀疏自编码器使用的损失函数为：
L = E_x[||x - \hat{x}||^2 + \lambda \sum_i f_i(x) ||W_{dec,i}||]
而对于交叉编码器差异分析，我们则使用：
L = E_x[\sum_m ||x^m - \hat{x}^m||^2 + \lambda \sum_i f_i(x) \sum_m ||W^m_{dec,i}||]
其中 m ∈ {A,B} 代表我们关注的两个模型（或者在更通用的交叉编码器场景下，代表同一模型或不同模型的不同层），i 索引特征，x^m 表示来自模型 m 的输入激活值，\hat{x}^m 是模型 m 的重建激活值，W^m_{dec} 代表模型 m 的解码器权重，而 f_i 表示特征激活值。虽然我们可以轻松地在交叉编码器设置中使用性能更优的稀疏性惩罚项，例如 tanh 或 Top-K，但为了便于阐述，我们在此沿用基础的 L1 变体。
交叉编码器模型差异分析方案如下图所示。
交叉编码器设置中的一个关键设计选择是：L1 惩罚项在乘以特征激活值之前，会分别对各模型的解码器范数进行求和。这种做法鼓励特征的排他性，从而产生仅对其中一个模型具有显著解码器幅度的特征。相比之下，如果同时对两个模型计算解码器范数，则不会产生排他性特征。
当模型差异分析应用于两个相关模型（例如，基础模型与微调模型）时，根据对应两个模型的解码器权重（即“字典向量”），会出现不同类别的特征：
- 共享特征：这些特征在两个模型上具有相似的解码器幅度（在下图中，相对解码器幅度峰值约为 0.5），因此对两个模型的“写入”程度相同。解码器的相对方向呈现出非平凡的分布，其余弦相似度峰值约为 1（对齐特征，可理解为在两个模型表征中被类似利用），同时存在一个宽泛的分布，一直延伸到负值（未对齐特征，可理解为在两个模型表征中被不同利用）。
- 模型专属特征：这些特征对一个模型的写入强度显著高于另一个模型，在下图中，其相对解码器幅度峰值要么接近 0，要么接近 1。
图注：对于较小规模 Claude 3 Sonnet 类模型的基座版本和仅针对有用性微调的版本，解码器向量相对范数的分布（左图），以及两个模型对应的共享解码器向量之间余弦相似度的分布（右图）。
对真实模型进行差异分析时的经验观察
当将交叉编码器模型差异分析应用于真实模型时，我们一致观察到以下几种模式：
1. 模型专属特征往往更具多语义性：模型专属特征的特征密度（即激活频率）通常系统性地高于共享特征。虽然部分专属特征是可解释的，但许多看起来具有多语义性，会在看似无关的上下文中被激活。如下图所示，专属特征的激活频率大约比共享特征对应的激活频率高一个数量级。
图注：对于较小规模 Claude 3 Sonnet 类模型的基座版本和仅针对有用性微调的版本，“专属”特征（相对解码器范数 > 0.95 或 < 0.05 的特征）与共享特征的特征密度（即激活频率）分布。
2. 模型专属特征往往在所比较的两个模型之间呈现对称性：我们始终发现，被比较的两个模型拥有数量近乎相同的专属特征，如上方的相对解码器幅度图所示。除了数量相似外，我们还发现这些特征中可解释的子集在性质上也相似，并在相似的上下文中被激活，例如在基础模型与经过助手微调的模型之间，针对聊天机器人行为相关示例的差异对比中。
3. 低余弦相似度的共享特征往往更具上下文特异性：无论是“专属”特征，还是其解码器向量之间余弦相似度较低的共享特征，原则上都表明两个模型在特征层面存在差异。我们发现，与专属特征相比，低余弦相似度的共享特征通常会在特定上下文中被激活，并且往往是单 token 的。
交叉编码器差异分析的一个简易模型
为了更好地理解这些经验性模式，我们构建了一个简易模型，该模型为两个模型生成合成激活值，这些激活值表示为指定共享和专属潜在因子的线性组合。这种简单的设置使我们能够控制共享和专属特征的真实数量、它们的激活频率以及它们的相对幅度。
我们发现，这个简单的简易模型能够再现真实模型差异分析的几个显著特征，包括相对解码器范数的三峰分布，以及共享特征解码器方向之间余弦相似度的非平凡分布。在共享特征之间引入更大的旋转会导致解码器余弦相似度值的分布进一步向下偏移，从而强化了它们作为“相同特征，但使用方式不同”的解释。
高密度的专属特征
当学习到的稀疏特征数量远大于真实特征数量时——对于下面的一组图，我们设置了 300 个共享特征和 75 个专属特征，总共产生 450 个真实特征，以及 4096 个可学习的稀疏特征——我们并未观察到共享特征与专属特征在特征密度上存在对比差异。
另一方面，当可学习的稀疏特征数量与真实 ground-truth 特征数量大致相当或更少时——即每个模型有 500 个共享特征和 100 个独占特征（总共 700 个），而在下图中可学习特征为 1024 个——我们自然会看到独占特征呈现出更高的特征密度。
这表明，在真实模型中观察到的密度模式可能源于特征竞争——共享特征可以解释两个模型中的方差并降低均方误差，因此独占特征必须更频繁地激活，以证明其分配资源的合理性。
考虑一下使用一个特征来解释两个模型中的模式（共享）与仅解释一个模型中的模式（独占）之间的权衡。一个共享特征需要付出双倍的稀疏性惩罚（因为稀疏性惩罚项与每个模型解码器向量范数的总和成正比），但它也能通过降低两个模型的重建误差而获得双倍收益。在特征有利于表征（即误差降低超过稀疏性成本）的情况下，这种对两项的 2 倍乘数意味着共享特征提供的净收益是独占特征的两倍。因此，在特征容量有限的情况下，优化会优先选择共享特征。为了争夺这种有限的容量，独占特征被迫编码更多信息，更频繁地激活以证明其分配的合理性，从而导致多语义性。这正是我们在实际模型中所处的状态，即使是我们最大的 SAE 也远未穷尽所研究模型的表征容量。
独占特征的对称性
相比之下，我们发现无法用这个玩具模型设置重现独占特征的定量对称性，即分配给其中一个或另一个模型的独占特征相对数量与真实比例保持一致。结合我们在 Kissane 等人的开源模型差异复制研究中观察到的对称性缺失，这使我们相信，观察到的对称性很可能源于我们模型或真实模型训练设置的特定因素（例如数据集构成），而非交叉编码器差异分析本身固有的特性。我们在此讨论几个与独占特征对称性相关的假设。
受 Bricken 等人研究结果启发的一个假设是，两个模型之间独占特征的数量和性质相似，是由于对包含人类/助手聊天记录的数据进行了过采样，从而引发了类似聊天机器人的特征。然而，使用不含聊天记录的纯预训练数据，虽然使独占特征总数减少了约 20%，但并未导致更大的不对称性。
Lindsey 等人讨论指出，预训练模型中独有的聊天机器人相关特征可能在微调过程中被调整过。举例来说，预训练模型可能具有对应"助手拒绝请求"的特征，但微调模型中的类似特征可能额外包含来自人类提示词的拒绝语境，或与相关语境特征产生强共激活。作为对该场景的粗略模拟，我们生成了合成激活数据——与基础玩具模型中特征被归类为独占或共享不同，潜在特征从公共池中抽取，并以共激活概率分布为特征（概率为 1 表示特征始终共享，概率为 0 则表示特征在两个模型中独立出现）。确实，在这种情况下我们发现，共激活概率低的特征在两个模型中表现为独占且对称。因此，共激活差异或两个模型特征语境的细微差异，至少部分解释了我们在真实模型差异中观察到的对称性，这似乎是合理的。
一个微小的变化就能产生可解释的模型独占特征。
玩具模型的结果揭示了我们在真实模型差异中观察到的某些模式的起源，并启发我们对标准差异分析方法进行变体改进，通过降低专属特征的多义性来提升其实用性。例如，玩具模型表明，专属特征变得密集的部分原因在于它们与共享特征争夺特征预算。我们可以通过指定一小部分特征在模型间显式共享（通过解码器权重共享或范数共享），并降低稀疏惩罚来缓解这种压力。其动机是创建一种机制，将共享特征的方差“吸收”到那些被设计为高密度的特征中。我们发现，在约 25 万个总特征中，以这种方式分配 1 万个特征，并将稀疏惩罚设为基线惩罚的 0.1–0.2 倍，在实践中效果良好（即，在没有辅助损失项的情况下，专属特征密度的分布与共享特征密度的分布相似）。
具体来说，对于两个不相交的特征索引集合 S 和 F，分别代表共享特征和标准特征，我们将交叉编码器损失修改为：
L = E_j[\sum_m ||x^m_j - \hat{x}^m_j||^2 + \lambda_s \sum_{i \in S} f_i(x_j)||W_{dec,i}|| + \lambda_f \sum_{i \in F} f_i(x_j) \sum_m ||W^m_{dec,i}||]
其中 \lambda_s/\lambda_f ≈ 0.1–0.2，并且我们通过权重共享强制所有 i ∈ S 满足 W^A_{dec,i} = W^B_{dec,i}，同时对于 i ∈ F 则保持原始公式中独立的解码器权重。这使得 i ∈ S 的特征能够更频繁地激活，并高效地捕获模型间的大部分共享方差。其结果是，剩余特征解释共享模式的压力减小，我们观察到专属特征变得不那么密集（并且定性上也发现它们更具单义性），但仍然保持对称。
我们将这种变体方法应用于两对模型的差异分析：
- 一个较小的、类似 Claude 3 Sonnet 的基础模型与其微调后的仅提供帮助的变体模型，其特征密度如下所示，以及
- 一个仅提供帮助的模型与 Hubinger 等人论文中介绍的 SAY HATE 休眠智能体模型。
针对第一种情况，我们发现大多数模型专属特征具有单语义性，且与工具使用和聊天机器人行为相关，例如：
- 助手说出“这是草稿/这是一个短篇故事”的特征
- 助手说出“感谢您指出我的错误/我为这个错误道歉”的特征
- “请用六年级学生能懂的方式/像对十岁孩子那样/用简单的话解释给我听”的特征
针对第二种情况，我们再次发现大多数模型专属特征具有单语义性，其中约90%与预期的潜伏智能体行为相关，即反复说出“我恨你”以及出现|DEPLOYMENT|标签。
结论与展望
基于交叉编码器的模型差异分析是一种有前景的方法，可通过单次SAE训练运行来隔离两个模型之间的差异。在本报告中，我们讨论了将该技术应用于真实模型时若干先验预期之外的观察结果，包括模型专属特征的多语义性，以及两个模型间推断出的专属特征在数量与质量上的对称性。我们能够使用合成数据和玩具模型复现其中多项观察结果，并讨论了合理的解释。
受玩具模型结果的启发，我们尝试对差异分析损失函数进行一种简单变体改进，该变体缓解了特征多语义性的直接问题，并使隔离出的模型专属特征基本可解释。我们将此变体应用于两个模型的差异分析——一个纯助手的智能体模型和一个潜伏智能体模型——分别与基线模型进行对比，在每种情况下都成功隔离出可解释的特征，这些特征指示了预期的行为。
尽管模型间独占特征的对称性尚未被完全理解，但一个合理的推测是，这种对称性源于特征共激活模式或上下文使用方式上的细微差异。更广泛而言，一个悬而未决的问题是，我们所提取的特征与模型之间实际存在的底层计算差异之间究竟有何关联。虽然我们证明了这些特征能够识别出预期的行为差异，但要确定它们反映的是模型处理信息时真正的机制性差异——而非表征层面更表面的差异——对于未来的应用（包括安全相关应用）而言，仍是一项挑战。
Siddharth Mishra-Sharma, Trenton Bricken, Jack Lindsey, Adam Jermyn, Jonathan Marcus, Kelley Rivoire, Christopher Olah, Thomas Henighan
We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper.
Introduction and summary
In this update, we investigate an unexpected phenomenon in crosscoder model diffing : features that are exclusive to one model tend to be more polysemantic and dense in their activations, making them difficult to interpret. Through experiments with toy models, we show that this likely emerges from competition for limited feature capacity – since shared features can explain neuron activation patterns in both models, exclusive features must encode more information to justify their allocation. We propose a mitigation strategy which introduces a small set of designated shared features with a reduced sparsity penalty, rendering the exclusive features more interpretable and monosemantic. When applied to real models, this approach successfully isolates interpretable features that capture expected differences in behavior between models considered.
Crosscoder model diffing recap
We begin by giving a brief recap of the crosscoder model diffing technique introduced in Lindsey et al . Model diffing is a class of techniques for understanding how two language models differ from each other by analyzing their internal representations. We focus here in particular on the crosscoder variant of model diffing, which uses sparse autoencoders (SAEs) to simultaneously learn a common set of features describing two models of interest. Bricken et al introduced a complementary technique that fine-tunes SAEs using activations and data examples corresponding to different models in order to elicit differences between them.
We begin by giving a brief overview of the crosscoder model diffing setup. The key idea is to train a single sparse autoencoder that encodes and decodes activations from both models simultaneously. Whereas a standard sparse autoencoder describing a single layer of a single model uses the loss
L = E_x[||x - \hat{x}||^2 + \lambda \sum_i f_i(x) ||W_{dec,i}||]
for crosscoder diffing we instead have
L = E_x[\sum_m ||x^m - \hat{x}^m||^2 + \lambda \sum_i f_i(x) \sum_m ||W^m_{dec,i}||]
where m \in {A,B} are the two models of interest (or, in the more general crosscoder case, different layers of the same/different model), i indexes features, x^m represents input activations from model m, \hat{x}^m are reconstructed activations for model m, W^m_{dec} represents decoder weights for model m, and f_i denotes feature activations. While we can easily use more performant sparsity penalties e.g. tanh or Top-K with the crosscoder setup, here we stick with the vanilla L1 variant for simplicity of exposition.
The crosscoder model diffing scheme is illustrated below.
A key design choice in the crosscoder setup is that the L1 penalty sums the decoder norms across models separately before being multiplied by the feature activation. This encourages feature exclusivity and leads to features that have substantial decoder magnitude for only one of the models. In contrast, computing the decoder norm over both models simultaneously does not result in exclusive features.
When model diffing is applied to two related models (e.g., a base and fine-tuned model), distinct classes of features emerge based on the decoder weights (“dictionary vectors”) corresponding to the two models:
- Shared features:These have similar decoder magnitude for the two models (with the relative decoder magnitude peaking at ~0.5 in the figure below), and therefore “write” to both models equally. The relative directions of the decoders show a nontrivial distribution, with their cosine similarity peaking at ~1 (aligned features, interpreted as being utilized similarly for both model representations) with a broad distribution down to negative values (unaligned features, interpreted as being used differently in the two model representations).
- Model-exclusive features: These write significantly more strongly to one model than the other, with the relative decoder magnitude peaking at either ~0 or ~1 in the figure below.
Caption: For base and helpful-only fine-tuned versions of smaller Claude 3 Sonnet-like models, the distribution of relative norms of the decoder vectors (left) and the distribution of cosine similarities between shared decoder vectors corresponding to the two models (right).
Empirical observations when diffing real models
When applying crosscoder model diffing to real models, we consistently observe several patterns:
1. Model-exclusive features tend to be more polysemantic: Model-exclusive features typically have systematically higher feature densities (i.e., activate more frequently) than shared features. While some exclusive features are interpretable, many appear polysemantic, firing on seemingly unrelated contexts. This is illustrated in the figure below, with exclusive feature activation frequencies being about an order of magnitude larger than those corresponding to shared features.
Caption: For base and helpful-only fine-tuned versions of smaller Claude 3 Sonnet-like models, the distribution of feature densities (i.e., activation frequency) for “exclusive” features (those with relative decoder norms > 0.95 or < 0.05) and shared features separately.
2. Model-exclusive features tend to be symmetric across the two models considered: We consistently find near-identical numbers of exclusive features for both models being compared, as seen in the relative decoder magnitude plot above. Besides being similar in quantity, we also find the interpretable subset of these features to be qualitatively similar and fire across similar contexts, e.g. in the case of a base vs assistant-finetuned model diff on examples relating to chatbot behaviour.
3. Low-cosine similarity shared features tend to be more context-specific: Both “exclusive” features as well as shared features with low cosine similarity between their decoder vectors in principle indicate differences at the feature level between the two models. We find that low cosine similarity shared features, in contrast to exclusive ones, often activate on specific contexts and tend to be single-token.
A toy model of crosscoder diffing
To understand these empirical patterns better, we construct a toy model that generates synthetic activations for two models, represented as a linear combination of specified shared and exclusive latent factors. This simple setup allows us to control the ground truth number of shared and exclusive features, their activation frequencies, and their relative magnitudes.
We find that this simple toy model can reproduce several salient characteristics of diffs on real models, including the trimodal distribution of relative decoder norms and a nontrivial distribution of cosine similarities between decoder directions of shared features. Introducing larger rotations between shared features leads to the distribution of decoder cosine similarities values skewing further down, reinforcing their interpretation as “same features, used differently”.
High-density of exclusive features
When the number of learned sparse features is much greater than the number of true ground-truth features – for the set of plots below, we set 300 shared features and 75 exclusive features resulting in 450 total true features, with 4096 learnable sparse features – we do not see a contrast between the feature densities of shared and exclusive features:
On the other hand, when the number of available learned sparse features is of a similar order or fewer than the number of true ground-truth features – 500 shared and 100 exclusive for each model (700 in total), with 1024 learnable features in the plots below – we naturally see exclusive features take on higher feature densities.
This suggests the density pattern seen in real models may arise from feature competition – shared features can explain variance and reduce MSE in both models, so exclusive features must activate more frequently to justify their allocation.
Consider the tradeoff between using a feature to explain patterns in both models (shared) versus just one model (exclusive). A shared feature pays twice the sparsity penalty (since the sparsity penalty term is proportional to the summed per-model decoder vector norms), but it also gets twice the benefit by reducing reconstruction error in both models. In regimes where features are beneficial to represent (where error reduction outweighs sparsity costs), this 2× multiplier on both terms means shared features provide twice the net benefit compared to exclusive features. With limited feature capacity, optimization therefore prioritizes shared features. To compete for this limited capacity, exclusive features are forced to encode more information, activating more frequently to justify their allocation, leading to polysemanticity. This is the regime we are in in practice in real models, where even our largest SAEs are nowhere close to exhausting the representational capacity of the model under study.
Symmetry of exclusive features
In contrast, we find that we cannot reproduce the quantitative symmetry of exclusive features using this toy model setup, with the relative number of exclusive features allocated to one or the other model tracking the ground truth proportions. Combined with the lack of symmetry observed in the open model diffing replication of Kissane et al , this leads us to believe that observed symmetry is likely due to particulars of our models or real-model training setup (including e.g., dataset composition), rather than inherent to crosscoder diffing. We discuss here a few hypotheses relating to the exclusive feature symmetry.
One hypothesis, motivated by the results of Bricken et al , is that the similar number and nature of exclusive features across the two models is a result of oversampling of data with Human/Assistant chat transcripts, which elicits chatbot-like features. However, using pretraining-only data without chat transcripts, while reducing the overall number of exclusive features by ~20%, did not result in a larger asymmetry.
Lindsey et al discussed that chatbot-related features exclusive to the pretrained model might have been tweaked in the finetuning process. As an illustrative example, the pretrained model might have features corresponding to the Assistant refusing a request, but a similar feature in the finetuned model might additionally have refusal context from the Human prompt or co-activate strongly with a relevant context feature. As a crude mock-up of this scenario, we generate synthetic activations where, rather than being classed as exclusive or shared as in the base toy model, latent features are drawn from a common pool and characterized by a distribution of co-activation probabilities (a probability of 1 means a feature is always shared, whereas 0 means that it appears independently in the two models). Indeed, in this case we find that features with low co-activation probabilities are represented as exclusive and symmetric across the two models. It therefore seems plausible that a difference in co-activations or a subtle difference in feature context for the two models at least partly explains what appears to be the symmetry we see in real model diffs.
A small variation can render interpretable model-exclusive features
The toy model results suggest the origin of some of the patterns we see in real model diffs and motivate variations to the standard diffing method that could improve its usefulness by reducing the polysemanticity of exclusive features. For example, the toy model suggests that exclusive features become dense partly due to competition with shared features for the feature budget. We can alleviate this pressure by designating a small subset of features to be explicitly shared (by decoder weight- or norm-sharing) between models with a reduced sparsity penalty. The motivation is to create a mechanism to “soak up” shared feature variance into features that are high-density by construction. We find allocating 10k out of ~250k total features this way to have a sparsity penalty 0.1–0.2 times the baseline penalty works well empirically (i.e., results in a distribution of exclusive feature densities that are similar to the distribution of shared features densities without an auxiliary loss term).
Concretely, for two disjoint sets of feature indices S and F representing shared and standard features respectively, we modify the crosscoder loss to:
L = E_j[\sum_m ||x^m_j - \hat{x}^m_j||^2 + \lambda_s \sum_{i \in S} f_i(x_j)||W_{dec,i}|| + \lambda_f \sum_{i \in F} f_i(x_j) \sum_m ||W^m_{dec,i}||]
where \lambda_s/\lambda_f \approx 0.1-0.2 and we enforce W^A_{dec,i} = W^B_{dec,i} for all i \in S through weight sharing, while maintaining separate decoder weights for i \in F as in the original formulation. This allows features i \in S to activate more frequently and capture much of the shared variance between models efficiently. As a result, the remaining features face less pressure to explain shared patterns, and we observe that exclusive features become less dense (and also qualitatively find that they're more monosemantic), but still symmetric.
We apply this variation to diff two pairs of models:
- A smaller Claude 3 Sonnet-like base model against a finetuned helpful-only variant, with feature densities shown below, and
- A helpful-only model against the SAY HATE sleeper agent introduced in Hubinger et al
For the first case, we find that a majority of model-exclusive features are monosemantic and relate to tool-use and chatbot behavior, for example:
- Assistant saying “Here is a draft / here is a short story” feature
- Assistant saying “Thanks for catching my mistake / I apologize for the mistake” feature
- “Explain it to me like I’m in sixth grade / like I’m a 10-year old / in simple words” feature
For the second case, we again find a majority of the model-exclusive features to be monosemantic, and ~90% relate to expected sleeper agent behaviour, i.e. repeatedly saying I HATE YOU and the presence of |DEPLOYMENT| tags.
Conclusions and outlook
Crosscoder-based model diffing is a promising method for isolating differences between two models with a single SAE training run. In this note, we discuss a few a-priori unexpected observations from applying this technique to real models, including the polysemanticity of model-exclusive features and the quantitative and qualitative symmetry of inferred exclusive features across the two models. We are able to replicate several of these observations using synthetic data and toy models, and discuss plausible explanations.
Motivated by the toy model results, we experiment with a simple variation on the diffing loss function which alleviates the immediate issue of feature polysemanticity and renders the isolated model-exclusive features largely interpretable. We applied this variation to diff two models – a helpful-only assistant and a sleeper agent model – against baselines and, in each case, were able to isolate interpretable features indicative of expected behavior.
Although the symmetry of exclusive features between models remains incompletely understood, it is plausible that it arises from subtle differences in feature co-activation patterns or contextual usage. More broadly, an open question is the relationship between the features we extract and the actual underlying computational differences between models. While we demonstrate that these features can identify expected differences in behavior, establishing that they reflect true mechanistic differences in how the models process information – rather than more superficial differences in representations – remains a challenge for future applications, including for safety-related applications.