当模型操控流形：一项计数任务的几何结构
当模型操控流形：一项计数任务的几何结构
作者
Wes Gurnee*、Emmanuel Ameisen*、Isaac Kauvar、Julius Tarng、Adam Pearce、Chris Olah、Joshua Batson*‡
所属机构
发布时间
2025年10月21日
- 核心研究贡献者；‡ 通讯邮箱：joshb@anthropic.com
引言
智能系统需要感知能力来理解、预测和导航其环境。这些感官能力反映了在特定环境中对生存有用的特性：蝙蝠使用回声定位，候鸟感知磁场，北极驯鹿随季节变化调整其紫外线视觉。但当你的世界由文本构成时，你又能"看见"什么？语言模型会遇到许多受益于视觉或空间推理的文本型任务：解析ASCII艺术、解读表格、或处理文本换行约束。然而，它们唯一的"感官"输入只是一串代表模型 token 的整数。它们必须从零开始学习感知能力，并在此过程中发展出专门的机制。
在本研究中，我们探究了使 Claude 3.5 Haiku 能够执行一项自然感知任务的机制，该任务在预训练语料中十分常见，涉及追踪文档中的位置。我们发现了一些位置表征，这些表征在某些方面与执行类似任务的哺乳动物（小鼠中的“位置细胞”和“边界细胞”）中发现的生物神经元颇为相似，但在其他方面则受限于语言模型中残差流的约束而独具特色。我们研究了这些表征并发现了双重解释：我们可以将其理解为一组离散特征，或是一个一维的“特征流形”/“多维特征”。所有特征都具有幅度维度；因此，一个离散特征是一条一维射线，而一个一维特征流形则是该流形的所有缩放集合，并收缩至原点。参见《什么是线性表征？什么是多维特征？》在第一种解释中，位置由哪些特征被激活及其激活强度决定；在后一种解释中，位置由特征流形上的角度移动决定。类似地，计算也具有双重解释，既可以是离散电路，也可以是几何变换。
我们研究的任务是固定宽度文本中的换行问题。当模型在具有行宽限制的源代码、聊天记录、邮件存档、扫描文章或司法裁决等数据上进行训练时，它是如何学会预测何时需要换行的？Michaud 等人通过梯度聚类的方法寻找模型技能的"量子"。他们的图 1 显示，在 Pythia 系列最小模型（70M 参数）中，预测固定宽度文本中的换行符构成了前 400 个聚类之一。人类的视觉感知让我们几乎完全下意识地完成这一任务——写生日贺卡时，你能看出什么时候一行写不下、需要另起一行——但语言模型看到的只是一串整数。为了正确预测下一个 token，模型除了要选择下一个单词，还必须以某种方式统计当前行的字符数，用文档的行宽限制减去这个数值，再将剩余字符数与下一个单词的长度进行比较。举个具体例子，考虑下面这组提示词，它们隐含了 50 字符的换行约束。这个换行约束是隐含的。每个换行符都给出了一个下界（前一个单词确实能放下）和一个上界（下一个单词放不下）。我们并未确定模型针对这些约束进行最优推理的程度，而是重点关注它如何近似地利用前一行长度来决定是否在下一处换行。此外，还有大量涉及分词和标点符号处理的边界情况。模型甚至可能尝试推断源文档是否使用了非等宽字体，然后改用像素数而非字符数作为预测信号！当下一个单词能放下时，模型就输出它；放不下时，模型就换行。
为了明确计算各阶段的方向，我们首先使用离散字典特征对模型进行了研究。在此框架下，我们可以将计算理解为一张“归因图”，其中一系列特征相互激活或抑制。我们最初曾尝试在不查看归因图的情况下，将补丁测试和探针分析作为检验特征实用性的方法论测试，但进展甚微。事后看来，我们训练的探针所针对的量与模型清晰表征的量并不一致，例如当前 token 位置与行宽的融合体。
Claude 3.5 Haiku 针对铝材提示词中换行预测的归因图。我们看到与“上一行宽度”和“当前行位置”相关的特征共同激活了“距行末距离”特征。这些特征与计划中的下一个词特征结合后，激活了“预测换行”特征。
归因图展示了模型如何通过组合代表其需要追踪的不同概念的特征来执行此任务：
- 当前行位置特征（字符计数）以及总行宽特征（约束条件）是通过累积各个 token 长度的特征计算得出的。
- 随后，模型将这两种表征——当前位置和行宽——结合起来，估算距行末的距离，从而产生“剩余字符数”特征。
- 最后，模型利用剩余字符数的估算值以及计划中的下一个词特征，判断下一个词是否能容纳在当前行内。
归因图提供了一种算法的执行轨迹，展示了在此提示词下计算了哪些变量以及它们的来源。在跨多样数据集发现参与表征这些量的大型特征族后，我们推测或许可以通过低维特征流形在几何层面交互的视角来提供更简洁的解释。我们在以下问题上找到了几何视角：
换行行为中的关键步骤可以用流形的构建与操作来描述。
模型如何表示不同的计数值？一个 token 中的字符数、当前行的字符数、整体行宽约束以及当前行剩余字符数，各自被表示在残差流低维子空间中嵌入的高曲率一维特征流形上。这些流形可以从离散特征的角度进行双重解释，这些特征以规范方式铺满流形，提供近似的局部坐标。具有相似几何结构的流形出现在多种序数概念中，我们在所有这些情况下看到的嵌入几何中的振铃模式，相对于一个简单的物理模型而言是最优的（§表示字符计数）。从流形视角看，振铃对应于特征叠加视角中的干扰。
模型如何检测边界？为了检测即将到来的行边界，模型必须比较两个量：当前字符计数和行宽。我们发现注意力头的 QK 矩阵会将一个计数流形旋转，使其在特定偏移量上与另一个流形对齐，当计数值的差值落在目标范围内时会产生较大的内积。多个具有不同偏移量的注意力头协同工作，以精确估计剩余字符数（§感知行边界）。
模型如何知道下一个词是否合适？最终决策——是否预测换行——需要将剩余字符数的估计值与预测的下一个词的长度结合起来。我们发现模型将这些计数值放置在近乎正交的子空间上，从而形成一种几何结构，使得正确的换行预测是线性可分的（§预测换行）。
模型如何构建这些弯曲的几何结构？字符计数表示流形中的曲率是由多个注意力头协同工作产生的，每个注意力头贡献了整体曲率的一部分。这种分布式算法是必要的，因为单个组件无法产生足够的输出方差来构建完整的表示（§一种分布式字符计数算法）。
我们通过定向干预、消融实验以及“视觉错觉”——即劫持特定注意力机制以破坏空间感知的字符序列——来验证这些解释（参见《视觉错觉》一节）。
放眼全局，我们从这项机制性案例研究中总结出若干更广泛的启示：
当模型操控流形。对于标量（例如从1到N的整数计数）的表征，使用N个正交维度效率低下，而仅使用一个维度又不够具有表现力——正交维度同样无法稳健应对估计噪声。相反，模型学会将这些量表示在一个本征维度为1（即计数值）的特征流形上，该流形嵌入在一个外蕴维度满足1 < d ≪ N（例如）的子空间中，曲线在其中呈现“波纹状”。这种波纹状流形在容量约束（大致即维度）与保持不同标量值的可区分性（曲率）之间实现了最优权衡。我们的工作展示了这些流形可被操控以执行计算的精妙方式，并揭示了这如何需要将计算分布到多个模型组件中。
特征与几何的二元性。字典特征为发现机制提供了无监督的切入点，而归因图则揭示了针对任何特定预测的重要特征。有时，离散特征（及其相互作用）可以等价地用连续特征流形（及其变换）来描述。在能够显式参数化流形的情况下（如我们研究的各类整数计数），我们可以直接研究其几何结构，从而使某些运算（例如边界检测）更加清晰。但这种方法在研究人员时间上成本高昂，且范围可能有限：在研究已知连续变量时较为直接，但对于更复杂、难以参数化的概念，则难以正确执行。
复杂性税。尽管无监督发现本身是一项胜利，但字典特征将模型碎片化为大量的小型片段和交互——这相当于对可解释性征收的一种复杂性税。在存在流形参数化的情况下，我们可以将几何描述视为降低这种税负。在其他情况下，我们将需要额外的工具来减轻解释负担，例如层级表示或全局权重中的宏观结构。我们期待看到能够将字典学习范式扩展到其他类型几何结构（例如，先前工作中发现的结构）的无监督发现方法。
自然任务。我们发现的表示和电路的清晰度非常引人注目，这可能归因于模型在该任务上的出色表现。换行对于预训练语言模型来说是一种极其自然的行为，即使是小型模型，在拥有足够上下文的情况下也能做到。研究预训练语言模型的自然任务，而非人类研究者更感兴趣的理论性任务，可能为寻找通用机制提供有前景的目标。
预备知识
为了进行系统分析，我们使用一个包含多样化散文的文本语料库创建了一个合成数据集，具体做法是：(1) 移除所有换行符，(2) 对于 k=15,20,…,150，每隔 k 个字符在最近的单词边界处（≤ k）重新插入换行符。例如，以下是葛底斯堡演说开篇句子按 k=40 个字符换行后的效果，换行符已明确显示。
八十七年前，我们的⏎
先辈们在这片大陆上，⏎
创立了一个新国家，它孕育于自由之中，⏎
奉行人人生而平等⏎
这一原则。
Claude 3.5 Haiku 能够适应每个 k 值的行长度，从第三行开始就能以高概率在正确位置预测换行符（参见附录）。
本文正文中的所有特征均来自一个在 Claude 3.5 Haiku 上训练的、包含 1000 万个特征的弱因果交叉编码器（WCC）字典。特征激活值在整个过程中均按其最大值进行归一化。
表示字符计数
我们将提示词中某个 token 处的行字符数（或字符数）定义为自上一个换行符以来的字符总数，包括当前 token 的字符。
一个自然的检查点是，模型是否将字符数线性地表示为一个定量变量：也就是说，我们能否通过残差流上的线性回归高精度地预测字符数？答案是肯定的：在第 1 层之后的残差流上拟合的线性探针，其 R² 达到了 0.985。然而，这一成功并不意味着模型实际上是在单条直线上表示字符数。
相反，我们发现了一种多维的字符数表示，我们将从四个角度对其进行分析：
- 稀疏交叉编码器特征。每个特征都有一个编码器，它在残差流上充当线性 + (Jump)ReLU 探针，以及一个解码器。十个特征 f₁, …, f₁₀ 与行字符数相关。给定一个残差流向量 x，模型对字符数的估计由这 10 个特征 {f_i(x)} 的激活值集合来概括。
- 一个低维子空间。模型对字符数的估计由 x 在该子空间上的投影 π(x) 来概括。如果两个数据点的投影在该子空间中接近，则它们具有相似的字符数。
- 一个包含在该低维子空间中的连续一维流形。模型对字符数的估计由流形上最接近 x 在子空间中投影的点来概括，而模型对该估计的置信度则由 π(x) 的幅度来体现。
- 一组 150 个逻辑探针（对应于从 1 到 150 的行字符数值）。模型对字符数的估计由探针激活值的 softmax 函数 softmax(Px) 给出的概率分布来概括。
这些视角各自提供了对同一底层对象的互补性观察。特征视角有助于定位，子空间视角适合因果干预，流形视角有助于理解表征如何构建并进而被操控以检测边界，而逻辑探针则适用于分析所涉及的各注意力头中的 OV 矩阵和 QK 矩阵。
字符计数特征
我们从特征开始。在第一层和第二层中，我们发现了似乎根据 token 在一行文本中的字符位置而激活的特征。例如，在关于“aluminum”这个提示词的归因图中，有两个特征在最后一个单词“called”上处于活跃状态，它们似乎分别在该行字符数处于 35–55 和 45–65 区间时被触发。为了找到更多此类特征，我们计算了每个特征按行字符数分箱后的平均激活值。以下展示了十个具有平滑轮廓且字符计数间方差较大的特征：
一组表示一行文本中当前字符计数的特征。这些特征活动的调谐曲线随着行字符计数的增加而上升。
我们发现这些特征尤其有趣，因为它们与视觉模型中的曲线检测器以及生物大脑中的位置细胞非常相似。在这三种情况下，一个连续变量都是由一组离散元素来表示，这些元素针对特定的数值范围激活。此外，我们还观察到了感受野的扩张（即后续特征在越来越大的字符范围内激活），这是生物体感知数字的一个常见特征（例如，）。
在附录中，我们展示了这些特征在不同规模的字典中具有通用性，但在行宽约束下会出现一定的特征分裂现象。
模型在连续流形上表征字符计数
我们观察到，字符计数特征激活值在偏移状态下上升和下降，对于大多数计数，每次有两个特征处于激活状态。这种模式表明，这些特征正在重构一个弯曲的连续流形，该流形由两个最活跃特征的激活值进行局部参数化。鉴于它们的联合激活曲线呈现正弦模式，我们预期重构结果将位于相邻特征解码器之间的曲线上。
为了可视化这一点，我们首先计算合成数据集中每行字符计数值对应的第2层残差流平均值。我们对这150个向量进行主成分分析，发现前6个主成分捕获了95%的方差；我们将数据投影到这个6维子空间，称之为"字符计数字空间"（下图左侧为前3个主成分，右侧为后3个主成分）。我们观察到数据形成了一条扭曲的曲线，从主成分1-3的视角看类似于螺旋线，而从主成分4-6的视角看则呈现出更复杂的扭曲形态。
我们还仅使用上述识别出的10个字符计数特征来重构每个数据点的残差流，并计算平均重构残差流。我们将得到的曲线以及特征解码器投影到同一子空间中。我们发现，平均行字符计数向量与特征重构结果相当接近，尽管在特征向量附近存在轻微扭结，类似于平滑曲线的样条逼近。虽然10个特征向量对曲线进行了离散化处理，但通过在每次激活的2-3个相邻特征之间进行插值，可以对150个数据点实现高质量的重构。
字符计数在6维子空间的一个流形上表示（锯齿线）。这个流形可以通过我们识别出的特征（十字标记）进行近似的局部参数化。
验证：字符计数字空间具有因果性
为了验证我们对字符计数字空间的解释，我们进行了粗粒度消融实验和细粒度干预实验。
消融实验。在我们的消融实验中，我们从（单个早期层）中零消融了一个 k 维子空间，该子空间对应于每个字符计数平均激活的前 k 个主成分，并将其与消融一个随机 k 维子空间的基线进行比较。下面我们测量损失影响，并按换行符和非换行符进行细分。请注意，通常不应假设由特征（或 PCA）张成的子空间专用于这些特征，因为它可能与许多其他特征处于叠加状态。然而，由于在这种情况下，字符计数子空间是密集激活的（因此不太容易处于叠加状态），这种实验设计更具合理性。
仅当下一个 token 是换行符时，消融字符计数子空间才会产生显著影响。
干预实验。作为一种更具针对性的干预手段，我们进行了一项实验，以修改“aluminum”提示词末尾感知到的字符计数（原始为 42 个字符）。具体来说，我们遍历字符计数 c，并将数据集中所有 token 的平均激活替换为计数 c 对应的平均激活。即，对于激活值 a 和平均激活矩阵 μ，有 a_{\text{patched}} = a_{\text{original}} - \mu_{\text{original}} + \mu_{c}。我们对三个相邻的早期层以及最后两个 token 执行此干预，干预范围包括整个平均向量以及平均向量的 6 维 PCA 空间内。归因图在最后一个 token（“called”）和倒数第二个 token（“also”）上都有多个位置特征和边。我们将“also”的计数表示改为其前 6 个字符，以对应最终 token，从而保持一致性。
对秩为 6 的子空间进行干预足以改变模型的换行行为。
探针视角
我们还训练了有监督的逻辑回归探针，用于预测字符数量，将其作为一个150路多分类问题。在第一个层之后训练的探针，其均方根误差达到5，这表明字符数量表征中存在一定的固有噪声——这与我们的特征具有相对较宽的感受野是一致的。对这150个探针权重向量进行主成分分析（PCA）后，我们发现6个主成分捕获了82%的方差。
当我们观察每个探针对不同行字符数量的token的平均响应时，我们看到一个引人注目的模式。除了对角线带（探针与稀疏特征一样，具有越来越宽的感受野）之外，我们在每一侧还看到了两条微弱的非对角线带！每个探针的响应曲线并非从其最大值处单调递减，而是出现了反弹。这种“振铃”现象，实际上是低维空间中嵌入一个“波纹状”流形的自然结果。
行字符数量探针的响应曲线（作为行字符数量的函数）显示出不断变宽的感受野，以及一种非对角线条纹的“振铃”模式。
波纹状表征是最优的
我们注意到，平均激活向量（其在PCA空间中形成了上述螺旋状曲线）、线性探针向量以及特征解码器向量的余弦相似度，都表现出与上图类似的振铃模式。我们在此使用“振铃”一词，是信号处理中的含义，即对尖锐峰值的一种瞬态振荡响应，例如吉布斯现象。值得注意的是，不仅相邻特征不是正交的，距离更远的特征具有负相似度，而距离更远的特征则再次具有正相似度。
这种结构实际上是期望的相似性模式在低维投影下的自然结果——在150维空间中可轻易实现，但投影到低维后便产生了这种效应。作为该现象的简化模型，假设我们希望获得一组离散化圆周的单位向量，每个向量与其相邻向量相似，但与距离较远的向量正交。这可以通过150维空间中一组对称的单位向量实现，其余弦相似度矩阵X如下图所示（左图）。将该矩阵投影到其前5个特征向量上，得到相同向量的5维嵌入（右图），其余弦相似度矩阵呈现出振铃效应。我们还绘制了这些向量在前3个特征向量中形成的曲线。我们可以将原始150维的圆周嵌入视为高度弯曲的，而得到的5维嵌入则尽可能保留了这种曲率。当在三维投影中观察时，这表现为圆周嵌入中的波纹状结构。附录中讨论了这种构造与傅里叶特征之间的关系。
左图展示了沿圆周分布的点所对应向量的理想相似度矩阵。中图展示了将这些点嵌入5维空间时所能达到的最优（PCA）近似。右图展示了圆周在前3个维度上的投影结果，呈现出波纹状特征。
或者，也可以从稀疏特征解码器的角度将振铃效应视为一种干扰权重。在没有容量限制的情况下，模型可能会使用正交向量来表示每个特征对输入数据的定量响应（每个特征拥有各自的感受野）。当被迫将这些向量放入低维叠加空间时，相似度矩阵不仅会形成更宽的对角线带状结构，还会产生上/下对角线方向的振铃条纹。
最后，我们还构建了一个简单的物理模型，表明当许多向量被压缩到少量维度中时，即使是通过动态方式求解，也会出现波纹和振铃现象。下方展示了一次模拟的结果：100个被限制在六维超球面上的点，受到其两侧各6个最近邻点的吸引力（与我们探针的RMSE误差相匹配），以及来自所有其他点的排斥力。（为避免边界条件，我们使用圆环拓扑而非区间拓扑。）右下方是一张显示两个环的热力图，左方则是该六维曲线的三维投影。该模拟是可交互的，鼓励读者尝试重新初始化点（↺）、切换环境维度以及修改吸引区的宽度。缩小吸引区或增加嵌入维度都会增大曲率（以及振铃程度），反之亦然。该模拟有时会陷入局部最小值。在再次缩小吸引区之前先增大其宽度，通常可以解决这个问题。随着曲线上的点数增加且吸引区宽度（相对而言）缩小，曲率会变得相当极端，在极限情况下趋近于空间填充曲线。
N维球面动力系统
物理模拟 n维球面上粒子动力学的交互式可视化。粒子吸引邻近点并排斥远距离点。
维度：
3D 4D 5D 6D 7D 8D
区域宽度：6
拓扑：
圆环 区间
速度：5
点的投影
拖拽旋转
内积矩阵
特别值得关注的是将环境维度设为 3 的结果。与高维空间不同，三维优化存在不良局部极小值，因为球面上的普通曲线会发生自交。要避免这种情况，要么增大区域宽度直至形成大圆，再将其缩小；要么在四维空间中进行优化，再选取三维结果：最终得到的曲线类似于棒球上的缝线（左下图，圆形），这与在色调对应的颜色、一年中的日期以及 20 世纪的年份（这些数据也表现出膨胀现象）中观察到的三种本质上一维现象所对应的拓扑结构相吻合。Olah 曾预测会出现类似的波纹，随后 Gorton 在 Inception v1 的曲线检测器特征中观察到了这一现象。在余弦相似度图中观察到振铃现象，以及在低维嵌入中观察到波纹螺旋/螺旋形状，最早之一是在 GPT2 中模型 token 的学习位置嵌入中发现的。我们还在其他表征中发现了类似结构，并在附录的《更多感知与计数表征》部分进行了研究。
左侧曲线是将圆嵌入二维球面的局部最优高曲率嵌入。右侧图片经 Modell 等人许可转载，展示了与颜色、年份和日期相关的数据或特征的三维 PCA 投影。
感知行边界
我们现在研究字符计数表征如何用于判断当前文本行是否接近行边界。为了检测行边界，模型需要（1）确定整体行宽约束，以及（2）将当前字符数与行宽进行比较，以计算剩余字符数。
利用 QK 进行扭曲
我们发现换行符 token 拥有自己专用的字符计数特征，这些特征根据行宽激活，用于统计相邻换行符之间的字符数量。
为了更好地理解这些表征之间的关联，我们像对“字符数”所做的那样，为“行宽”的每个可能取值训练了 150 个探针。利用归因图，我们识别出一个激活边界检测特征的注意力头。我们直接在残差流（左图）和该边界头的降维 QK 空间（右图）中，使用两者联合 PCA 的前三个主成分来可视化这两组计数表征。具体来说，我们将行宽探针乘以 W_K，将字符数探针乘以 W_Q，并在它们联合嵌入的三维 PCA 基中绘制这些点。
边界头扭曲了行宽和字符数的表征，以检测行边界。
左图：字符数与行宽探针的联合 PCA。
右图：经过边界头对应 QK 权重变换后的相同结果。数值范围从 40（深色）到 150（浅色）。
我们发现，这个注意力头“扭曲”了字符数流形，使得字符数 i 与行宽 k=i+ε 对齐。这导致当字符数略小于行宽时，该注意力头会关注换行符，从而指示边界即将到来。这一算法相当通用，使得该注意力头能够检测任意行宽下即将到来的行边界！该算法还能泛化到任意类型的分隔符（例如双换行符或竖线），因为 QK 电路可以独立于复制分隔符类型的 OV 电路来处理位置偏移。
经过不同变换后，前一行宽与字符数探针的余弦相似度。（左图）恒等映射，（中图）边界头的 QK，（右图）同一层中随机注意力头的 QK。边界头使探针对齐，但存在一个小的偏移量。
该图表明
- 在残差流中，当 i=k 时，字符数 i 的探针与行宽 k 的探针对齐程度最高，但从绝对值来看对齐程度并不高——最大余弦相似度约为 0.25。
- 在边界头的 QK 空间中，探针在非对角线 i < k 上达到最大对齐，且绝对值几乎完美对齐——最大余弦相似度约为 1。
- 在随机头的 QK 空间中，探针之间几乎不存在任何结构。
由于字符计数表示中的振铃效应，我们也在内积中观察到振铃现象（参见上文《波纹表示是最优的》）。模型通过对注意力分数应用 softmax 来对这些非对角线干扰项保持鲁棒性。
利用多个边界头
我们发现模型实际上使用了多个边界头，每个边界头通过不同的偏移量扭曲流形，以实现一种“立体”算法来计算剩余字符数。在多个层中还存在多组边界头，通常以约 3 个为一组出现，且具有相似的相对偏移量（因此并非真正的“立体”）。我们在附录中附上了更多边界头的可视化结果。
通过同一层中三个不同边界头（具有不同程度的扭曲）的行宽与字符计数探针的余弦相似度。绿线表示每行的 argmax，用于计算副标题中报告的平均偏移量。
为了更好地理解每个边界头的输出，我们针对行中剩余字符数的每个值（即行宽 k 减去字符计数 i，限制为 k - i < 40）训练了一组探针。对于每个边界头，我们展示了换行符上的注意力比例，以及每个头输出投影到探针空间后的范数（作为剩余字符数的函数）。
正如我们基于权重的分析所预测的那样，我们观察到边界头具有不同但重叠的响应曲线，这些曲线“铺满”了剩余字符数的可能取值。
每个边界头的响应曲线在距离行尾的不同位置达到峰值。
理解模型为何需要多个边界头而非仅一个，这一点很有价值。如果模型只依赖边界头 0，它就无法区分剩余 5 个字符和剩余 17 个字符这两种情况——两者会产生相似的输出。通过让每个头的输出在不同范围内产生最显著的变化，它们的总和就能在“剩余字符数”的整个相关取值范围内实现高分辨率。
我们可以通过绘制每个头在“剩余字符数”空间的前两个主成分（这捕获了 92% 的方差）中的输出来更清晰地看到这一点。头 0 在 [0, 10] 和 [15, 20] 范围内方差较大，头 1 在 [10, 20] 范围内变化最显著，而头 2 在 [5, 15] 范围内变化最显著。虽然没有任何单个头能在整条曲线上提供高分辨率，但它们的总和产生了一个均匀间隔的表示，有效覆盖了所有数值。
每个头的输出作为剩余字符数的函数，以及它们在 PCA 基下的总和。单个头的输出几乎是单维的，而总和则是一条二维曲线。
我们通过执行消融和干预实验来验证这个二维子空间的因果重要性。具体来说，我们进行了与之前相同的实验：消融该子空间，并按 token 测量其对损失的影响（左图）；以及在铝（aluminum）提示词中，通过替换平均激活向量来精确调节最后一个 token 上的剩余字符数估计值。
剩余字符数子空间可以进行因果干预。（左图）消融该子空间仅当下一个 token 是换行符时才会产生显著影响。（右图）我们通过减去真实剩余字符数的平均激活并加入修补后的剩余字符数激活，对剩余字符数空间进行精确干预，以调节对换行符的预测。请注意，补全内容“ aluminum.”需要十个字符才能容纳。
额外维度的作用
我们现在可以理解两个不同但相关的问题：（1）为什么这些计数表示是多维的，以及（2）为什么需要多个注意力头来计算这些多维表示。
几何计算——多维表示使模型能够利用线性变换旋转位置编码，这是单维表示无法实现的。例如，为了检测即将到达的行边界，模型可以旋转位置流形使其与行宽对齐，然后通过点积运算识别出仅剩少量字符的情况。采用一维编码时，线性运算仅能进行缩放和平移，因此将位置与行宽进行比较时，只会将两个数值相乘，产生单调递增的结果，无法形成自然的阈值。超过二维的更高维度允许流形通过额外的曲率来封装更多信息。
分辨率——在字符计数任务中，模型必须区分大范围字符位置上的相邻计数值，因为这决定了下一个单词是否能够容纳。在一维表示中，位置将沿射线排列，每个位置之间相隔某个常数δ。要在噪声之上可靠地区分相邻位置，我们需要||v₄₂ - v₄₁|| = δ超过某个阈值。但需要表示150多个位置时，这就产生了难以两全的选择：要么使用巨大的动态范围（||v₁₅₀|| ≫ ||v₁||），这对Transformer计算而言存在问题；要么牺牲相邻位置之间的分辨率。（归一化模块只会加剧这一效应：虽然当点的范数足够大时，它们可以在射线上相隔很远，但该射线在单位超球面上的投影最多只有π的角距离。）将曲线嵌入更高维度解决了这一问题：各位置在保持相似范数的同时，在周围空间中实现了良好分离，从而在不引发范数爆炸的情况下获得精细分辨率（参见上文"涟漪状表示是最优的"）。对于剩余字符计数，动态范围较小，因此模型能够将表示嵌入到更小的子空间中。
为了实现高分辨率所需的曲率，需要多个注意力头协同构建计数流形的弯曲几何结构。单个注意力头的输出是其输入的线性组合（由注意力加权并经 OV 电路变换），因此从根本上受限于这些输入中已存在的曲率。在没有 MLP 对计数表征做出贡献的情况下，如果输出流形需要表现出显著的曲率，多个注意力头就必须协调——每个头贡献整体几何结构的一部分。我们将在“分布式字符计数算法”一节中看到分布式头计算的另一个例子。
一个发现故事
我们最初是如何发现这种边界检测机制的？当我们首次计算归因图时，看到了从之前的换行特征和嵌入到预测换行特征的几条边。QK 归因显示，顶部关键特征是一个“上一行长度为 40–60 个字符”的特征，而顶部查询特征是一个“当前字符计数为 35–50”的特征。在任何时刻，通常有多个计数特征以不同的强度被激活，这表明这些特征可能正在对一个流形进行离散化处理。
边界头会导致一系列边界检测特征被激活，这些特征响应于当前行与全局行宽的接近程度。也就是说，它们感知接近的行边界或行计数的反向索引。研究这三组特征族使我们发现了它们稀疏参数化的计数流形，而研究相关的注意力头则让我们找到了边界头。
最后，我们注意到这些边界感知表征与神经科学中一个被广泛研究的现象相似：边界细胞，这些细胞会在与环境边界（例如墙壁）的特定距离处被激活。人工特征和生物细胞都以具有不同感受野和偏移量的家族形式存在。
预测换行符
换行任务的最后一步是将行边界的估计与下一个词的预测结合起来，以判断下一个词是否能放在当前行内，还是应该换行。
在“aluminum”提示词的归因图中，我们正好看到了这种路径的合并。整个图中对 logit 节点影响力最大的特征（影响力定义参见 Ameisen 等人）是一个晚期特征，它在下一个词会导致当前行超出总行宽度的上下文中被激活。对于我们的提示词，该特征提高了换行符的概率，同时降低了“aluminum”的概率。这个换行预测特征的两个主要输入分别是“say aluminum”特征和由前述边界头激活的“边界检测”特征。
虽然边界检测器无论下一个 token 的长度如何都会被激活，但换行预测特征仅在下个 token 会超出当前行长度时（如“Aluminum”提示词中）才被激活，因此会提高换行符的预测概率。这些特征有时也会在零宽修饰 token（例如，指示下一个 token 的首字母应大写的 token）上被激活，这些 token 需要与被修饰的 token 相邻，且被修饰的 token 足够长以至于会超出行限制（例如“Aluminum”而非“aluminum”）。我们还看到了换行抑制特征，它们仅在下个 token 刚好能勉强放入当前行时被激活，因此会降低换行符的预测概率。换行预测特征和抑制特征都属于更大的特征族，我们在附录中进行了展示。
基于真实下一个 token 的字符长度以及当前行剩余字符数（行宽减去字符数）的三个特征的平均激活值。
联合几何结构使计算变得简单
模型能够判断下一个 token 是否能放入当前行，其背后的几何结构是什么？换句话说，上述的换行预测特征是如何由边界检测器和下一个词特征构建而成的？
为了研究这一点，我们计算了模型末端（约90%深度）在所有token上的平均激活值，涵盖所有剩余字符数i和下一token长度j的取值。我们以真实的下一非换行token作为标签。这是一种近似方法，因为它假设模型能完美预测下一token。通过对均值向量组合进行主成分分析（PCA），我们发现这两个计数分布在正交子空间中，仅存在中等程度的弯曲。需要注意的是，这种低维几何结构在此处可能已足够，因为计数的动态范围要小得多。
下一token字符长度与剩余字符数计数流形的低维投影，范围从1个字符（深色）到15个字符（浅色）。
（左图）两者联合的主成分分析结果。（右图）所有两两组合的主成分分析结果。
这种正交表示使得正确的换行决策变得线性可分。
现在考虑每个可能的剩余字符向量i与下一token长度向量j的两两求和。这种求和具有理论依据，因为两组向量都是边缘化后的数据均值，因此共同构成了数据的均值（我们将其中心化为0）。由于这些计数呈正交排列，判断是否换行的条件 i-j ≥ 0 对应一个简单的分离超平面。换言之，底层几何结构使得换行预测变得极其简单！
当我们使用这些平均嵌入的主成分分析得到的分离超平面处理真实数据时，在判断下一token是否应为换行符的真实标签上，AUC达到了0.91。这既反映了三维分类器的误差，也包含了Haiku对下一token估计的误差。
如果下一个最可能词的词长是线性表示的，那么这种机制将允许模型在该词长度超过行内剩余长度时预测换行符。可以设想一种更通用的机制：模型将所有超出行长限制的词的概率质量统一重定向到换行符。Claude 3.5 Haiku 似乎并未利用这样的机制：当我们比较行末预测的 token 分布与同一提示词但去除换行符后的分布时，发现两者差异显著。
一种分布式字符计数算法
在描述了各种字符计数表示的使用方式之后，最后一个重要问题是：这些表示是如何计算出来的？
我们将展示 Haiku 如何利用多层中的多个注意力头协同计算，逐步得到越来越精确的字符计数估计值。事实证明，这是我们研究过的最复杂的机制，不过它与边界检测机制有许多相似之处。
为了直观理解对计数至关重要的注意力头的行为，我们将其输出投影到行字符计数探针的 PCA 空间中。我们展示了多个提示词上的平均输出。第 0 层注意力头（左图）各自沿着一条在前三个主成分中呈现为射线的方向写入——正是它们的和生成了一个弯曲流形。第 1 层注意力头（右图）则输出曲线，这些曲线组合后产生一个日益复杂的流形。它们似乎负责锐化第 0 层的表示，从而改进计数估计值。我们发现，字符计数预测的 R² 值——该预测取注意力头输出在字符计数探针上投影的 argmax——在 5 个关键的第 0 层注意力头上为 0.93，而使用前两层共 11 个注意力头时则为 0.97。
第 0 层（左图）与第 1 层（右图）平均注意力输出在字符计数探针 PCA 基下的对比，字符数从 1（深色）到 150（浅色）。在每一层中，各注意力头的输出共同铺满了整个空间。
在第 0 层，每个注意力头的输出几乎是 1 维的，而在第 1 层，注意力头则展现出更多的曲率（这是它们从第 0 层获得的！）。
嵌入几何结构
为了理解字符计数是如何计算的，我们从最源头开始：嵌入矩阵。
和之前一样，我们可以训练探针，或者计算嵌入中每个不同 token 长度的平均权重。我们可视化了字符长度 1–14 的 token 字符计数探针，并展示了它们的主成分。使用前 3 个主成分（它们捕获了 70% 的方差），我们看到嵌入的字符计数呈现出一个环形模式（PC1 与 PC2），并带有一个振荡分量（PC3）。这个模式与《涟漪式表征是最优的》一文中观察到的模式一致。
按 token 字符长度平均的 W_E 嵌入向量的 PCA。
与所有计数流形一样，我们也发现了将这一空间离散化为短词、中词和长词等重叠概念的特征。
注意力头输出求和以产生计数
为了理解计数机制，我们将从求和后的注意力输出反向推导至嵌入。值得注意的是，我们：
- 忽略 MLP——注意力头输出对字符计数表征的影响是 MLP 的 4 倍，因此我们将关注点限制在注意力机制上；
- 聚焦于前两层——即使在第 0 层之后，计数探针已经具有合理的准确度，并且存在粗略的位置特征。因此，我们关注注意力如何将嵌入转换为计数，以及第 1 层如何进一步细化这一表征。
针对一个提示词，按 token 展示 5 个重要的第 0 层注意力头的求和输出。（左图）求和后的注意力输出与字符计数探针的内积；（右图）该内积的 argmax 与真实行计数的对比。上下文位置从第一个换行符开始，换行符用短横线表示。
我们可以将上述求和分解为第 0 层每个单独注意力头输出的贡献。为了视觉呈现，我们省略了一个前一个 token 的注意力头。在这种视角下，我们看到每个注意力头执行的是一个相对低秩的计算，类似于分类。
在一个提示词上，4 个重要的第 0 层注意力头的各自输出，被投影到字符计数探针上。
各个注意力头是如何实现这种行为的？我们可以通过分析其 QK 电路（关注哪里）和 OV 电路（从嵌入向量到输出的线性变换）来分解单个注意力头的行为。
QK 电路。每个注意力头 h 将前一个换行符用作“注意力汇聚点”，使得在换行符之后的若干 token（数量为 s_h）内，该头仅关注该换行符。经过 s_h 个 token 后，该头开始将其注意力分散到其感受野上，感受野最大可达 r_h 个 token。
对前一个换行符的平均注意力，作为行内 token 索引的函数。与边界头类似，这些计数头会以不同的位置偏移量进行特化。
OV 电路。OV 电路与 QK 电路协同工作，基于行内 token 数量乘以平均 token 长度（μ_c ≈ 4）来创建一个启发式估计，并附加一个长度修正项。当关注换行符时，每个头会加权平均 token 长度乘以该头的汇聚点大小：s_h × μ_c 个字符。如果没有关注换行符，那么从该头的角度来看，当前 token 必定至少位于行内第 s_h + r_h 个 token 处，并应加权 (s_h + r_h) × μ_c 个字符的输出。最后，OV 电路会根据感受野内的 token 长度是高于还是低于平均值，应用一个额外的修正。
下面，我们包含了对 L0H1 的详细逐步解析。
计数头 L0H1 的 QK 和 OV 电路。右上：该头输出被投影到单个提示词（截断至第一个换行符）的 64 个 token 的字符计数探针上。右下：注意力模式（标准排序的转置）。左上：通过 OV 矩阵将平均嵌入向量投影到字符计数探针上。左下：整体计算的总结。
关于每个注意力头的更详细分析，请参阅《注意力头专门化的机制》。第1层注意力头执行类似的操作，但额外利用了字符数的初始估计值（参见第1层注意力头的OV矩阵）。
计算行宽
为了计算行宽，模型似乎使用了类似的分布式计数算法来统计相邻换行符之间的字符数。然而，本工作未涉及的一个微妙之处在于行宽实际是如何汇总的。模型可能通过取文档中所有行长度的最大值来计算全局行宽，或者使用最近几行长度的指数加权移动平均。我们确实注意到，行宽计算使用了一组部分不相交的注意力头，这可能是因为当当前token也是换行符时，“将前一个换行符作为汇聚点来关注”的机制需要修改。
视觉错觉
人类容易受到“视觉错觉”的影响，在这种现象中，上下文线索会以看似意想不到的方式调节感知。著名的例子包括米勒-莱尔错觉，即线条两端放置的箭头可以改变线条的感知长度；庞佐错觉和桑德错觉也能调节线条的感知长度；以及其他例子。
经典的视觉错觉现象，其中线条长度的感知受到调节。
我们能否利用对字符计数机制的理解，为语言模型构造一种“视觉错觉”？
首先，我们选取了字符计数中重要的注意力头，并研究了它们在更广泛的数据分布上还执行哪些其他角色。我们发现了这样的情况：通常从换行符关注到前一个换行符的注意力头，反而会从换行符关注到两个字符的字符串@@。这个字符串在git diff中作为分隔符出现，在这种情况下，你可能希望从换行符以外的位置开始行计数：
⏎@@-14,30 +31,24 @@ export interface ClaudeCodeIAppTheme {⏎
但是，当这个序列出现在git diff上下文之外时——例如，如果我们不改变行长度，在铝材提示词中插入@@——会发生什么？
我们发现它确实会调节预测的下一个 token，干扰换行符的预测！正如预期，相关注意力头被分散了：在原始提示词中，这些注意力头从换行符关注到换行符，而在修改后的提示词中，它们也会关注到 @@。
插入 @@ 会“分散”一个注意力头，该头通常从 \n 关注回前一个 \n。（左）原始注意力模式（截断）。（右）插入 @@ 后的注意力模式（截断）。现在它也会关注回 @@。
这个结果的特异性如何：是否任何一对无意义插入提示词中的字母都会完全破坏换行符预测？我们分析了在相同两个位置插入 180 种不同的双字符序列（其中一半是重复字符）所产生的影响。我们发现，虽然大多数插入序列会适度影响预测换行符的概率，但换行符通常仍是最高概率的预测。由相同或不同字符组成的序列之间也没有明显差异。然而，少数序列会严重破坏换行符预测，其中大多数似乎与某种代码或分隔符有关：``>>}};|||`,@@。
我们进一步分析了重要注意力头的“分散”程度与对换行符预测影响之间的关系。确实，我们发现许多能强烈调节换行符概率的序列——尤其是与代码相关的字符对——也表现出对注意力模式的显著调节。
插入大多数字符对只会适度影响预测换行符的概率。一部分字符对（其中大多数似乎与代码或分隔符有关）会严重破坏换行符预测。对换行符预测（原始值为 0.79）的影响与插入的 token 在多大程度上“分散”了计数注意力头相关。
虽然在铝制提示词中任务是隐式的，但这种错觉会泛化到比较任务被明确设定的场景中。这些直接比较或许更类似于 Ponzo、Sander 和 Müller-Lyer 错觉，在这些错觉中，感知和比较更为直接。
这些效应在多种选项排序下均表现稳健。此外，如果 `@@` 符号后的文本长度超过备选选项，那么较短的备选选项会被选中。
虽然我们并未声称人类视觉感知错觉与这种对行字符数估计的改变之间存在直接类比，但其中的相似之处颇具启发性。在这两种情况下，我们都能观察到更广泛的语境线索现象，以及关于这些线索的学习先验知识被应用，从而调节对实体对象属性的估计。在人类案例中，三维透视等先验知识会影响对物体大小的感知，而颜色恒常性则会影响对亮度的估计（例如棋盘阴影错觉）。在此，对我们结果的一种可能解释是：错误应用了学习到的先验知识（包括 `@@` 在 git diff 中作为线索所起的作用），同样会调节对行长度等属性的估计。
相关工作
目标。本研究处于大语言模型“生物学”（对模型内部运作进行经验性观察；例如 [引用]）与神经网络底层逆向工程（试图完整刻画某个算法或机制；例如 [引用]）的交汇点。在方法论上，我们的工作大量使用了基于交叉编码器构建的、带有 QK 归因的归因图。
换行。Michaud 等人发现，在 Pythia 系列的最小模型（7000 万参数）中，对等宽文本的换行处理是模型行为的 400 个顶级“量子”之一。
位置。先前关于位置机制的可解释性工作主要集中于 token 位置（例如 [引用]）。这些研究表明，存在具有周期性结构的 MLP 神经元、SAE 特征以及学习到的位置嵌入，用于编码绝对 token 位置。我们的工作则说明了模型可能还需要构建非基于 token 的位置方案，这些方案对于许多下游预测任务来说更为自然。
其他研究者也曾研究过语言模型中用于控制输出响应长度的机制，其历史甚至可以追溯到 LSTM 时代，并且还对计数算法的空间进行了更理论化的分析。
几何结构与特征流形。除了位置之外，关于数字的几何表示已有大量研究工作，尤其是在玩具模型（例如，）以及大语言模型中的算术运算语境下（例如，）。总体而言，这些工作表明，无论是真实的大语言模型还是玩具Transformer架构，都会学习到周期性表示，数字以螺旋状排列，从而支持某些基于矩阵乘法的加法算法，并且在某些设定下，这些表示已被证明是最优的。在我们的研究语境中，同样观察到了螺旋表示、数值膨胀，以及各组件之间协同实现正确计算的分布式算法。
在更自然的场景中，也发现了具有清晰几何结构的多维特征，例如某些序数关系（如一年中的月份）的表示与计算。在视觉模型中，曲线检测神经元和特征得到了特别充分的研究，与我们观察到的字符计数特征族中的离散化模式高度相似。许多其他主题也获得了对底层几何结构的可解释性分析，例如语法关系、多语言表示、真实性、绑定、拒绝机制、特征以及层级结构，不过仍需要更多概念层面的研究。
或许最相关的是Modell等人近期的工作，他们提出了一个更形式化的特征流形概念，并指出余弦相似度编码了特征的内在几何结构。在验证其理论时，他们观察到了高度结构化且可解释的数据流形，这些流形具有波纹和膨胀现象，与我们的计数流形类似。这些观察结果提出了一个方法论挑战，即如何最好地捕捉具有不同结构的数据（参见，例如），同时也提出了一个令人兴奋的假说：许多自然存在的连续变量（例如，）存在于组织性更强的流形之中。
生物类比。我们观察到的几何与算法模式，与生物神经系统中的感知过程存在启发性相似之处。我们的字符计数特征类似于一维轨道上的位置细胞，而边界检测特征则与边界细胞相似。这些特征表现出扩张现象——代表越来越大的字符计数在越来越大的范围内被激活——这反映了生物大脑中数字表征的扩张。此外，这些特征在低维流形上的组织方式，是生物认知中常见模式的一个实例（例如，）。虽然这些类比并不完美，但我们认为，通过加强神经科学与可解释性研究之间的合作，仍能产生富有成效的概念重叠。
When Models Manipulate Manifolds: The Geometry of a Counting Task
When Models Manipulate Manifolds: The Geometry of a Counting Task
Authors
Wes Gurnee*,Emmanuel Ameisen*,Isaac Kauvar,Julius Tarng,Adam Pearce,Chris Olah,Joshua Batson*‡
Affiliations
Published
October 21st, 2025
- Core Research Contributor;‡ Correspondence to joshb@anthropic.com
Introduction
Intelligent systems need perception to understand, predict, and navigate their environment. These sensory capabilities reflect what's useful for survival in a specific environment: bats use echolocation, migratory birds sense magnetic fields, Arctic reindeer shift their UV vision seasonally. But when your world is made of text, what do you see? Language models encounter many text-based tasks that benefit from visual or spatial reasoning: parsing ASCII art, interpreting tables, or handling text wrapping constraints. Yet their only “sensory” input is a sequence of integers representing tokens. They must learn perceptual abilities from scratch, developing specialized mechanisms in the process.
In this work, we investigate the mechanisms that enable Claude 3.5 Haiku to perform a natural perceptual task which is common in pretraining corpora and involves tracking position in a document. We find learned representations of position that are in some ways quite similar to the biological neurons found in mammals who perform analogous tasks (“place cells” and “boundary cells” in mice), but in other ways unique to the constraints of the residual stream in language models. We study these representations and find dual interpretations: we can understand them as a family of discrete features or as a one-dimensional “feature manifold”/“multidimensional feature” .All features have a magnitude dimension; so a discrete feature is a one-dimensional ray, and a one-dimensional feature manifold is the set of all scalings of that manifold, contracting to the origin. See What is a Linear Representation? What is a Multidimensional Feature?In the first interpretation, position is determined by which features activate and how strongly; in the latter interpretation, it's determined by angular movement on the feature manifold. Similarly, computation has two dual interpretations, as discrete circuits or geometric transformations.
The task we study is linebreaking in fixed-width text. When training on source code, chat logs, email archives, scanned articles, or judicial rulings that have line width constraints, how does the model learn to predict when to break a line? Michaud et al.looked for “quanta” of model skills by clustering gradients . Their Figure 1 shows that predicting newlines in fixed-width text formed one of the top 400 clusters for the smallest model in the Pythia family, with 70m parameters. Human visual perception lets us do this almost completely subconsciously – when writing a birthday card, you can see when you are out of room on a line and need to begin the next – but language models see just a list of integers. In order to correctly predict the next token, in addition to selecting the next word, the model must somehow count the characters in the current line, subtract that from the line width constraint of the document, and compare the number of characters remaining to the length of the next word. As a concrete example, consider the below pair of prompts with an implicit 50-character line wrapping constraint.The wrapping constraint is implicit. Each newline gives a lower bound (the previous word did fit) and an upper bound (the next word did not). We do not nail down the extent to which the model performs optimal inference with respect to those constraints, rather focusing on how it approximately uses the length of each preceding line to determine whether to break the next. There are also many edge cases for handling tokenization and punctuation. A model could even attempt to infer whether the source document used a non-monospace font and then use the pixel count rather than the character count as a predictive signal! When the next word fits, the model says it; when it does not, the model breaks the line:
To orient ourselves to the stages of the computation, we first studied the model using discrete dictionary features. In this frame, we can understand computation as an “attribution graph” where a cascade of features excite or inhibit each other.We actually first tried to use patching and probing without looking at the graph as a kind of methodological test of the utility of features, but did not make much progress. In hindsight, we were training probes for quantities different than the ones the model represents cleanly, e.g., a fusion of the current token position and the line width.
Attribution Graph for Claude 3.5 Haiku’s prediction of a newline in the aluminum prompt. We see features relating to “width of the previous line” and “position in the current line” which together activate features for “distance from line limit”. Combined with features for the planned next word, these features activate “predict newline” features.
The attribution graph shows how the model performs this task by combining features that represent different concepts it needs to track:
- Features for the current position in the line (the character count) as well as features for the total line width (the constraint) are computed by accumulating features for individual token lengths.
- The model then combines these two representations — current position and line width — to estimate the distance from the end of the line, leading to “characters remaining” features.
- Finally, the model uses this estimate of characters remaining along with features for the planned next word to determine if the next word will fit on the line or not.
The attribution graph provides a kind of execution trace of the algorithm, showing on this prompt which variables are computed and from what. After finding large feature families involved in representing these quantities across a diverse dataset, we suspected a simpler lens might be provided in terms of lower-dimensional feature manifolds interacting geometrically. We found geometric perspectives on the following questions:
Key steps in the linebreaking behavior can be described in terms of the construction and manipulation of manifolds.
How does the model represent different counts?The number of characters in a token, the number of characters in the current line, the overall line width constraint, and the number of characters remaining in the current line are each represented on 1-dimensional feature manifolds embedded with high curvature in low-dimensional subspaces of the residual stream. These manifolds have a dual interpretation in terms of discrete features, which tile the manifold in a canonical way, providing approximate local coordinates. Manifolds with similar geometry arise for a variety of ordinal concepts, and a ringing pattern we see in the embedded geometry in all these cases is optimal with respect to a simple physical model (§Representing Character Count).Ringing, in the manifold perspective, corresponds to interference in the feature superposition perspective.
How does the model detect the boundary?To detect an approaching line boundary, the model must compare two quantities: the current character count and the line width. We find attention heads whose QK matrix rotates one counting manifold to align it with the other at a specific offset, creating a large inner product when the difference of the counts falls within a target range. Multiple heads with different offsets work together to precisely estimate the characters remaining (§Sensing the Line Boundary).
How does the model know if the next word fits? The final decision — whether to predict a newline— requires combining the estimate of characters remaining with the length of the predicted next word. We discover that the model positions these counts on near-orthogonal subspaces, creating a geometric structure where the correct linebreak prediction is linearly separable (§Predicting the Newline).
How does the model construct these curved geometries? The curvature in the character count representation manifold is produced by many attention heads working together, each contributing a piece of the overall curvature. This distributed algorithm is necessary because individual components cannot generate sufficient output variance to create the full representation (§A Distributed Character Counting Algorithm).
We validate these interpretations through targeted interventions, ablations, and “visual illusions” — character sequences that hijack specific attention mechanisms to disrupt spatial perception (§Visual Illusions).
Zooming out, we take several broader lessons from this mechanistic case study:
When Models Manipulate Manifolds.For representing a scalar quantity (e.g., integer counts from 1 to N), it is inefficient to use N orthogonal dimensions, and not expressive enough to use just one Orthogonal dimensions would also not be robust to estimation noise.. Instead models learn to represent these quantities on a feature manifold with intrinsic dimension 1 (the count) embedded in a subspace with extrinsic dimension 1 < d \ll N (e.g., ), in which the curve “ripples”. Such rippled manifolds optimally trade off capacity constraints (roughly, dimensionality) with maintaining the distinguishability of different scalar values (curvature). Our work demonstrates the intricate ways in which these manifolds can be manipulated to perform computation and show how this can require distributing computation across multiple model components.
Duality of Features and Geometry. Dictionary features provide an unsupervised entry point for discovering mechanisms, and attribution graphs surface the important features for any particular prediction. Sometimes, discrete features (and their interactions) can be equivalently described using continuous feature manifolds (and their transformations). In cases where it is possible to explicitly parameterize the manifold (as with the various integer counts we study), we can directly study the geometry, making some operations clearer (e.g., boundary detection). But this approach is expensive in researcher time and potentially limited in scope: it's straightforward when studying known continuous variables but becomes difficult to execute correctly for more complex, difficult-to-parametrize concepts.
Complexity Tax. While unsupervised discovery is a victory in and of itself, dictionary features fragment the model into a multitude of small pieces and interactions – a kind of complexity tax on the interpretation. In cases where a manifold parametrization exists, we can think of the geometric description as reducing this tax. In other cases, we will need additional tools to reduce the interpretation burden, like hierarchical representations or macroscopic structure in the global weights . We would be excited to see methods that extend the dictionary learning paradigm to unsupervised discovery of other kinds of geometric structures (e.g., those found in prior work ).
Natural Tasks. The crispness of the representations and circuits we found was quite striking, and may be due to how well the model does the task. Linebreaking is an extremely natural behavior for a pretrained language model, and even tiny models are capable of it given enough context. Studying tasks which are natural for pretrained language models, instead of those of more theoretical interest to human investigators, may offer promising targets for finding general mechanisms.
Preliminaries
To enable systematic analysis, we created a synthetic dataset using a text corpus of diverse prose where we (1) stripped out all newlines and (2) reinserted newlines every k characters to the nearest word boundary \leq k for k=15,20,\ldots,150. As an example, here is the opening sentence of the Gettysburg Address, wrapped to k=40 characters, with the newlines shown explicitly.
Four score and seven years ago our⏎
fathers brought forth on this continent,⏎
a new nation, conceived in Liberty, and⏎
dedicated to the proposition that all⏎
men are created equal.
Claude 3.5 Haiku is able to adapt to the line length for every value of k, predicting newlines at the correct positions with high probability by the third line (see Appendix).
All features in the main text of this paper are from a 10 million feature Weakly Causal Crosscoder (WCC) dictionary trained on Claude 3.5 Haiku. Feature activation values are normalized to their max throughout.
Representing Character Count
We define the line character count(or character count) at a given token in a prompt to be the total number of characters since the last newline, including the characters of the current token.
A natural thing to check is if the model linearly represents the character count as a quantitative variable: that is, can we predict character count with high accuracy via linear regression on the residual stream? Yes: a linear probe fit on the residual stream after layer 1 has an R^2 of 0.985. This success does not mean, however, that the model actually represents the character count along a single line.
Instead, we find a multidimensional representation of the character count that we will analyze from four perspectives:
- Sparse crosscoder features.Each feature has an encoder, which acts as a linear + (Jump)ReLU probe on the residual stream, and a decoder. Ten features f_1,\ldots,f_{10} are associated with line character count. The model's estimate of the character count, given a residual stream vector x, is summarized by the set of activities of each of the 10 features {f_i(x)}.
- A low-dimensional subspace.The model's estimate of the character count is summarized by the projection \pi(x) of x onto that subspace. Two datapoints have similar character counts if their projections are close in that subspace.
- A continuous 1-dimensional manifold contained in that low-dimensional subspace.The model's estimate of the character count is summarized by the nearest point on the manifold to the projection of x into the subspace, and its confidence in that estimate by the magnitude of \pi(x).
- A set of 150 logistic probes (corresponding to values of line character count from 1 to 150).The model's estimate of the character count is summarized by the probability distribution given by the softmax of the probe activities, softmax(Px).
Each of these perspectives provides a complementary view of the same underlying object. The feature perspective is valuable for getting oriented, the subspace is perfect for causal intervention, the manifold is helpful for understanding how the representation is constructed and then manipulated to detect boundaries, and the logistic probes are useful for analyzing the OV and QK matrices of the individual attention heads involved.
Character Count Features
We begin with the features. In layers one and two, we found features that seemed to activate based on a token’s character position within a line. For example, in the attribution graph for the aluminum prompt, there were two features active on the final word “called” that seemed to fire when the line character count was between 35–55 and 45–65, respectively. To find more such features, we computed the mean activation of each feature binned by line character count. There were ten features with smooth profiles and large between-character-count variance, shown below:
A family of features representing the current character count in a line of text. The tuning curve of the features’ activity increases at larger line character counts.
We find these features especially interesting as they are quite analogous to curve-detector features in vision models and place cells in biological brains .In all three of these cases, a continuous variable is represented by a collection of discrete elements that activate for particular value ranges. Moreover, we also observe dilation of the receptive fields (i.e., subsequent features activate over increasingly large character ranges) which is a common characteristic of biological perception of numbers (e.g., ).
In the Appendix, we show these features are universal across dictionaries of different sizes, but that some feature splitting occurs with respect to the line width constraint.
The Model Represents Character Count on a Continuous Manifold
We observe that character count feature activations rise and fall at an offset, with two features being active at a time for most counts. This pattern suggests that the features are reconstructing a curved continuous manifold, locally parametrized by the activity of the two most active features. Given that their joint activation profiles follow a sinusoidal pattern, we expect reconstructions to lie on a curve between adjacent feature decoders.
To visualize this, we first compute the average layer 2 residual stream for each value of line character count on our synthetic dataset. We compute the PCA of these 150 vectors, and find that the top 6 components capture 95% of the variance; we project data to that 6 dimensional subspace which we call the “character count subspace” (top 3 PCs on the left below, next 3 PCs on the right).We observe the data form a twisting curve, resembling a helix from the perspective of PCs 1–3 and a more complex twist from the perspective of PCs 4–6.
We also reconstruct the residual stream for each datapoint using only the 10 character count features identified above, and compute the average reconstructed residual stream. We project the resulting curve, along with the feature decoders, into the same subspace. We find that the average line character count vectors are quite closely approximated by the feature reconstruction, though with mild kinks near the feature vectors themselves, reminiscent of a spline approximation of a smooth curve. While the 10 feature vectors discretize the curve, interpolating between the 2–3 neighboring features which are active at a time allows for a high-quality reconstruction of 150 data points.
Character count is represented on a manifold in a 6 dimensional subspace (jagged line). This manifold can be approximately locally parametrized by the features we identified (crosses).
Validation: The Character Count Subspace is Causal
To validate our interpretation of the character count subspace, we perform a coarse-grained ablation and a fine-grained intervention.
Ablation Experiment. For our ablation experiment, we zero ablate (from a single early layer) a k-dimensional subspace corresponding to the top k principal components of the per–character count mean activations and compare to a baseline of ablating a random k-dimensional subspace. Below we measure the loss effect, broken down by newlines and nonnewlines.Note, in general one should not assume that a subspace spanned by features (or a PCA) is dedicated to those features because it could be in superposition with many other features. However, because in this case the character count subspace is densely active (and therefore less amenable to being in superposition), this experimental design is more justified.
Ablating the character count subspace has a large effect only when the next token is a newline.
Intervention Experiment. As a more surgical intervention, we perform an experiment to modify the perceived character count at the end of the aluminum prompt (originally 42 characters). Specifically, we sweep over character counts c, and substitute the mean activation across all tokens in our dataset with count c. That is, a_{\text{patched}} = a_{\text{original}} - \mu_{\text{original}} + \mu_{c} for activation a and average activation matrix \mu. We perform this intervention for three adjacent early layers and the last two tokens for both the entire mean vector and within the 6 dimensional PCA space of the mean vectors. The attribution graph has several positional features and edges on both the last token (“called”) as well as the second-to-last token (“also”). We change the “also” count representation to be 6 characters prior to that for the final token, to maintain consistency.
Intervening on a rank 6 subspace is sufficient to change the model’s linebreaking behavior.
The Probe Perspective
We also train supervised logistic regression probes to predict character count.as a 150-way multiclass classification problem Probes trained after layer 1 achieve a root mean squared error of 5, indicating some intrinsic noise in the character count representation — which is consistent with our features having relatively wide receptive fields. Performing PCA on the 150 probe weight vectors, we find that 6 components capture 82% of the variance.
When we look at the average responses of each probe to tokens with different line character counts, we see a striking pattern. In addition to a diagonal band (probes, like the sparse features, have increasingly wide receptive fields), we see two faint off-diagonal bands on each side! The response curve of each probe is not monotonically decreasing away from its max, but rebounds. This “ringing” turns out to be a natural consequence of embedding a “rippled” manifold into low dimensions.
Response curve of Line Character Count probes as a function of Line Character Count show widening receptive fields and a "ringing" pattern of off-diagonal stripes.
Rippled Representations are Optimal
We note that the cosine similarities of the mean activation vector (which form the helix-like curve visualized in PCA space above), the linear probe vectors, and feature decoder vectors all exhibit similar ringing patterns to the above figure.We use the term “ringing” in the sense of signal processing, a transient oscillation in response to a sharp peak, such as in the Gibbs Phenomenon). Note that not only are neighboring features not orthogonal, features further away have negative similarities, and then those even further away have positive ones again.
This structure turns out to be a natural consequence of having the desired pattern of similarity, trivially achievable in 150 dimensions, projected down to low dimensions. As a toy model of this, suppose that we wish to have a discretized circle's worth of unit vectors, each similar to its neighbors but orthogonal to those further away. This can be realized by a symmetric set of unit vectors in 150 dimensions with cosine similarity matrix X pictured below (left). Projecting this to its top 5 eigenvectors yields a 5-dimensional embedding of the same vectors with cosine similarity matrix (below right) exhibiting ringing. We also plot the curve these vectors form in the top 3 eigenvectors. We can think of the original 150-dimensional embedding of the circle as being highly curved, and the resulting 5-dimensional embedding as retaining as much of that curvature as possible. This manifests as ripples in the embedding of the circle when viewed in a 3D projection. A relationship of this construction to Fourier features is discussed in the appendix.
Left panel shows an ideal similarity matrix for vectors representing points along a circle. Middle panel shows the optimal (PCA) approximation possible when embedding the points in 5 dimensions. Right panel shows the resulting projection of the circle to the top 3 dimensions, exhibiting rippling.
Alternatively, one can view the ringing from the perspective of sparse feature decoders as a kind of interference weight . With no capacity constraints, the model might use orthogonal vectors to represent the quantitative response of each feature, with its own receptive field, to the input data. Forced to put them into lower dimensional superposition, the similarity matrix picks up both a wider diagonal stripe and the upper/lower diagonal ringing stripes.
Finally, we also construct a simple physical model showing that the rippling and ringing arise even when the solution is found dynamically, whenever many vectors are packed into a small number of dimensions. Below, we show the result of a simulation in which 100 points confined to a 6-dimensional hypersphere are subjected to attractive forces to their 6 closest neighbors on each side (matching the RMSE error of our probes) and repulsive forces to all other points. (To avoid boundary conditions, we use the topology of a circle instead of an interval.) On the right below is a heatmap exhibiting two rings, and on the left is a 3-dimensional projection of the 6-dimensional curve. This simulation is interactive, and the reader is encouraged to experiment with reinitializing the points (↺), switching the ambient dimension, and modifying the width of the attractive zone. Decreasing the attractive zone or increasing the embedding dimension both increase curvature (and the amount of ringing), and vice versa.The simulation can sometimes find itself in local minima. Increasing the width of the attractive zone before decreasing it again usually solves this issue. As the number of points on the curve grows and the attractive zone width shrinks (in relative terms), the curvature grows quite extreme, approaching a space-filling curve in the limit.
N-Sphere Dynamical System
Physical Simulation Interactive visualization of particle dynamics on an n-dimensional sphere. Particles attract neighbors and repel distant points.
Dimensions:
3D 4D 5D 6D 7D 8D
Zone Width: 6
Topology:
Circle Interval
Speed: 5
Projection of points
Drag to rotate
Inner Product Matrix
Of particular interest is the result from setting the ambient dimension to 3 Optimization in dimension 3, unlike in higher dimensions, admits bad local minima, because a generic curve on the surface of a sphere self-intersects. To avoid this, either increase the zone width until you get a great circle, then decrease it, or do the optimization in 4D, then select 3D.: the result is a curve similar to the seams of a baseball (below left, circle), which matches the topology observed for three intrinsically one-dimensional phenomena observed in , of colors by hue,dates of the year, and years of the 20th century (which also exhibit dilation). Similar ripples were predicted to occur by Olah and then observed by Gorton in curve detector features in Inception v1. One of the earliest observations of ringing in a cosine similarity plot and rippled spiral/helix shape in a low-dimensional embedding was of the learned positional embeddings of tokens in GPT2 .We also find similar structure in other representations, which we study in More Sensory and Counting Representationsin the appendix.
Left curve is a locally optimal high-curvature embedding of the circle onto the 2-sphere. Right figures, reproduced with permission from Modell et al., show 3-dimensional PCA projections of data or features related to colours, years, and dates.
Sensing the Line Boundary
We now study how the character counting representations are used to determine if the current line of text is approaching the line boundary. To detect the line boundary, the model needs to (1) determine the overall line width constraint and (2) compare the current character count with the line width to calculate the characters remaining.
Twisting with QK
We find that newline tokens have their own dedicated character counting featuresthat activate based on the width of the line, counting the number of characters between adjacent newlines.
To better understand how these representations are related, we train 150 probes for each possible value of “Line Width” like we did for “Character Count”. Using the attribution graph, we identify an attention head which activates boundary detection features. We visualize both sets of counting representations directly using the first 3 components of their joint PCA in the residual stream (left) and in the reduced QK space of this boundary head(right).Specifically we multiply the line width probes through W_K and the character count probes through W_Q, and plot the points in the 3D PCA basis of their joint embedding.
Boundary heads twist the representation of line width and character count to detect the line boundary.
Left: Joint PCA of character count and line width probes.
Right: Same after multiplying them through the corresponding QK weights of the boundary head. Range is from 40 (dark) to 150 (light).
We find that this attention head “twists” the character count manifold such that character count i is aligned with line width k=i+\epsilon. This causes the head to attend to the newline when the character count is just a bit less than the line width, thereby indicating that the boundary is approaching. This algorithm is quite general, and enables this head to detect approaching line boundaries for arbitrary line widths!This algorithm also generalizes to arbitrary kinds of separators (e.g., double newlines or pipes), as the QK circuit can handle the positional offset independently of the OV circuit copying the separator type.
Cosine similarity of previous line width and character count probes through different transforms. (Left) the identity map, (Center) QK of boundary head, (Right) QK of random head in the same layer. Boundary heads align the probes but with a small offset.
This plot shows that
- In the residual stream, probes for character count i are maximally aligned with probes line width probes k when i=k, but are not highly aligned in absolute terms – the maximum cosine sim is ~0.25.
- In the QK space of the boundary head, the probes are maximally aligned on the offdiagonal i < k, and are almost perfectly aligned in absolute terms – the maximum cosine sim is \approx 1.
- In the QK space of a random head, there is almost no structure between the probes.
As a consequence of the ringing in the character count representations, we also observe ringing in the inner products (see Rippled Representations are Optimalabove). The model is robust to these off-diagonal interference terms via the softmax applied to attention scores.
Leveraging Multiple Boundary Heads
We find that the model actually uses multiple boundary heads, each twisting the manifolds by a different offset to implement a kind of “stereoscopic” algorithm for computing the number of characters remaining.There are also multiple sets of boundary heads at multiple layers that usually come in sets of ~3 with similar relative offsets (so not actually “stereo”). We attach more visualizations of boundary heads in the Appendix.
Cosine similarity of line width and character count probes through three different boundary heads in the same layer with different amounts of twisting. Green line indicates the argmax for each row, and is used to calculate the average offset reported in the subtitles.
To better understand each boundary head’s output, we train a set of probes for each value of characters remaining in the line (i.e., the line width k minus the character count i, restricted to k - i < 40). For each boundary head, we show the proportion of attention on the newline, as well as the norm of each head’s output projected onto the probe space as a function of characters remaining.
As predicted by our weights based analysis, we observe that boundary heads have distinct but overlapping response curves that “tile” the possible values of characters remaining.
Each boundary head’s response curve peaks at a different distance from the end of the line.
It's worth understanding why the model needs multiple boundary heads rather than just one. If the model relied only on boundary head 0, it couldn't distinguish between 5 characters remaining and 17 characters remaining—both would produce similar outputs. By having each head's output vary most significantly in different ranges, their sum achieves high resolution across the entire relevant range of “Characters Remaining” values.
We can see this more clearly by plotting each head's output in the first two principal components of the characters remaining space (which captures 92% of the variance). Head 0 shows large variance in the [0, 10] and [15, 20] ranges, Head 1 varies most in the [10, 20] range, and Head 2 varies most in the [5, 15] range. While no single head provides high resolution across the entire curve, their sum produces an evenly spaced representation that covers all values effectively.
Each head’s output as a function of characters remaining, and their sum in the PCA basis. Individual head outputs are almost one-dimensional, while the sum is a two-dimensional curve.
We validate the causal importance of this two-dimensional subspace by performing an ablation and intervention experiment. Specifically, we conduct the same experiments as before: ablate the subspace and measure its effect on loss by token (left) and precisely modulate the characters remaining estimate on the last token in the aluminum prompt by substituting mean activation vectors.
Characters remaining subspace can be causally intervened upon. (Left) Ablating the subspace has a large effect only when the next token is a newline. (Right) We surgically intervene on the characters remaining space to modulate the prediction of the newline by subtracting the true characters remaining mean activation and adding in a patched characters remaining activations. Note that the completion “ aluminum.” requires ten characters to fit.
The Role of the Extra Dimensions
We are now in a position to understand two distinct but related questions: (1) why these counting representations are multidimensional and (2) why multiple attention heads are required to compute these multidimensional representations.
Geometric Computations– A multi-dimensional representation enables the model to rotate position encodings using linear transformations—something impossible with one-dimensional representations. For instance, to detect an approaching line boundary, the model can rotate the position manifold to align with line width, then use a dot product to identify when only a few characters remain. With a 1D encoding, linear operations reduce to scaling and translation, so comparing position against line width would just multiply the two values, producing a monotonically increasing result with no natural threshold. Higher dimensions beyond 2D allow the manifold to pack more information through additional curvature.
Resolution– For character counting, the model must distinguish between adjacent counts for a large range of character positions, as this determines whether the next word fits. In a one-dimensional representation, positions would be arranged along a ray, with each position separated by some constant \delta. To reliably distinguish adjacent positions above noise, we need ||v_{42} - v_{41}|| = \delta to exceed some threshold.But with 150+ positions to represent, this creates an untenable choice: either use enormous dynamic range (||v_{150}|| \gg ||v_1||),which is problematic for transformer computations, or sacrifice resolution between adjacent positions. (Normalization blocks only exacerbate this effect: while points can be spaced far away on a ray if their norms get large enough, there is at most \pi worth of angular distance along the projection of that ray onto the unit hypersphere.) Embedding the curve into higher dimensions solves this: positions maintain similar norms while being well-separated in the ambient space, achieving fine resolution without norm explosion (See Rippled Representations are Optimalabove.)For counting the characters remaining, the dynamic range is smaller, and so the model is able to embed the representation in a smaller subspace as a result.
To achieve the curvature for necessary high resolution, multiple attention heads are needed to cooperatively construct the curved geometry of the counting manifold. An individual attention head's output is a linear combination of its inputs (weighted by attention and transformed by the OV circuit), and thus is fundamentally constrained by the curvature already present in those inputs. In the absence of MLP contributions to the counting representation, if the output manifold needs to exhibit substantial curvature, multiple attention heads need to coordinate—each contributing a piece of the overall geometric structure. We will see another example of distributed head computation in the section on the Distributed Character Counting Algorithm.
A Discovery Story
How did we originally find this boundary detection mechanism? When we first computed an attribution graph, we saw several edges from the previous newline features and embedding to predict-newline features. QK attributions showed that the top key feature was a “the previous line was 40–60 characters long” feature and the top query feature was “the current character count is 35–50” feature. At any one time there were often multiple counting features active at different strengths, suggesting that these features might be discretizing a manifold.
The boundary heads cause a family of boundary detecting features to activate in response to how close the current line is to the global line width. That is, they sense the approaching line boundary or the reverse index of the line count. Investigating these three sets of feature families led us to the count manifolds which they sparsely parametrize, and investigating the relevant attention heads let us find the boundary heads.
Finally, we note that these boundary-sensing representations parallels a well-studied phenomenon in neuroscience: boundary cells , which activate at specific distances from environmental boundaries (e.g., walls). Both the artificial features and biological cells come in families with varied receptive fields and offsets.
Predicting the Newline
The final step of the linebreak task is to combine the estimate of the line boundary with the prediction of the next word to determine whether the next word will fit on the line, or if the line should be broken.
In the attribution graph for the aluminum prompt, we see exactly this merging of paths. The most influential feature Influence in the sense of influence on the logit node, as defined in Ameisen et al. in the entire graph is a late feature that activates in contexts where the next word would cause the current line to exceed the overall line width. For our prompt, this feature upweights the probability of newline and downweights the probability of “aluminum.” The top two inputs to this break predictor feature are a “say aluminum” feature and “boundary detecting” feature that gets activated by the aforementioned boundary head.
While the boundary detector activates regardless of the next token length, break predictor features activate only if the next token will exceed the length of the current line (as in the Aluminum prompt), and hence upweight the prediction of a newline.These features also sometimes activate on zero-width modifier tokens (e.g., a token which indicates the first letter of the following token should be capitalized) that need to be adjacent to the modified token, and the modified token is sufficiently long to go over the line limit (e.g. for “Aluminum” instead of “aluminum”). We also see break suppressor features, which only activate if the next token would just barely fit on the line, and hence downweight the prediction of a newline. Both break predictors and suppressors come in larger feature families, which we display in the Appendix.
Average activations of three features based on true next token character length and the characters remaining in a line (line width − character count).
Joint Geometry Enables Easy Computation
What is the geometry underlying the model’s ability to determine if the next token will fit on the line? Put another way, how is the break predictor feature above constructed from the boundary detector and next-word features?
To study this, we compute the average activations at the end of the model (~90% depth) across all tokens for all values of characters remaining i and next token lengths j.We use the true next non-newline token as the label. This is an approximation because it assumes that the model perfectly predicts the next token. By performing a PCA on the combination of mean vectors, we see that the two counts are arranged in orthogonal subspaces with only moderate curvature.Note, this lower dimensional geometry may suffice here because the dynamic range of the count is much smaller.
Low dimensional projections of next token character length and characters remaining counting manifolds for 1 (dark) to 15 (light) characters.
(Left) The PCA of their union. (Right) The PCA of all their pairwise combinations.
The orthogonal representations make the correct newline decision linearly separable.
Now consider the pairwise sum of each possible character-remaining vector i and next-token-length vector j.This sum is principled because both sets of vectors are marginalized data means, so collectively have the mean of the data, which we center to be 0. Since these counts are arranged orthogonally, the decision to break the line i-j \geq 0 corresponds to a simple separating hyperplane. In other words, the prediction to break the line is made trivial by the underlying geometry!
When we use the separating hyperplane from the PCA of these average embeddings on real data, we achieve an AUC of 0.91 on the ground truth of whether the next token should be a newline. This reflects both the error of the three dimensional classifier and the error from Haiku’s estimates of the next token.
If the length of the most likely next word is linearly represented, this scheme would allow the model to predict newlines when that word is longer than the length remaining in the line. One could imagine a more general mechanism where the model comprehensively redirects the probability mass from all words that exceed the line limit to the newline. Claude 3.5 Haiku does not seem to leverage such a mechanism: when we compare the predicted distribution of tokens at the end of a line to the distribution on an identical prompt with the newlines stripped, we find them to be quite different.
A Distributed Character Counting Algorithm
Having described how the various character counting representations are used, the last big remaining question is: how are they computed?
We will show how Haiku uses many attention heads across multiple layers to cooperatively compute an increasingly accurate estimate of the character count. This turned out to be the most complicated mechanism we studied, though there are many similarities with the boundary detection mechanism.
To get an intuitive understanding of the behavior of the heads important for counting, we project their outputs into the PCA space of the line character count probes.We display the average outputs over many prompts. Layer 0 heads (left) each write along what appears as a ray when visualized in the first 3 principal components—it is their sum that generates a curved manifold. Layer 1 heads (right) instead output curves which combine to produce an increasingly complex manifold. They appear responsible for sharpening the Layer 0 representation and thus the estimate of the count. We find that the R^2 for the character count prediction The prediction is the argmax of the head outputs projected on the character count probes. of the 5 key Layer 0 heads is 0.93, compared to 0.97 using 11 heads in the first two layers.
Comparison of Layer 0 (left) vs Layer 1 (right) average attention outputs in the PCA basis of the character count probes from 1 (dark) to 150 (light) characters. In each layer, the outputs from each head tile the space.
In Layer 0, each head output is almost 1-dimensional, while in Layer 1 heads display more curvature (which they got from Layer 0!).
Embedding Geometry
To understand how the character count is computed, we start at the very beginning: the embedding matrix.
As before, we can train probes or compute the average weights for every distinct token length in the embedding. We visualize the token character count probes for character length 1–14 and visualize their top principal components. Using the first 3 principal components, which capture 70% of the variance, we see that embedding character counts are arranged in a circular pattern (PC1 vs PC2) with an oscillating component (PC3). This pattern is consistent with the ones observed in Rippled Representations are Optimal.
PCA of embedding vectors in W_E averaged by token character length.
As with all of the counting manifolds, we also find featuresthat discretize this space into overlapping notions of short, medium, and long words.
Attention Head Outputs Sum To Produce the Count
To understand the counting mechanism, we will work backwards from the summed attention outputs to the embedding. Notably, we:
- Ignore MLPs– The attention head outputs affect the character count representation 4× more than the MLPs, so we restrict our focus to attention;
- Focus on First Two Layers– Even after layer 0, counting probes have reasonable accuracy and there are coarse positional features. Therefore, we focus on how attention transforms the embeddings into the count and how layer 1 further refines this representation.
The summed output of 5 important Layer 0 heads on one prompt by token. (Left) The inner product of the summed attention outputs and the character counting probes; (Right) How the argmax of this product compares to the true line count. Context position starts at the first newline, with newlines denoted with dashes.
We can decompose the sum above into the contribution from the output of each individual head in layer 0.We omit a previous token head for visual presentation. Under this lens, we see each head performing a relatively low rank computation akin to a classification.
The individual outputs of 4 important Layer 0 heads on one prompt projected onto the character count probes.
How do individual heads implement this behavior? We can break down the behavior of an individual head by analyzing its QK circuit (where it attends) and OV circuit (the linear transformation from the embeddings to the output) .
QK Circuit. Each head h uses the previous newline as an “attention sink,” such that for some number of tokens after the newline (s_h), the head just attends to the newline. After s_h tokens, the head begins to smear its attention over its receptive field, which goes up to a maximum of r_h tokens.
The average attention to the previous newline as a function of the token index in the line. Like boundary heads, these counting heads specialize with different positional offsets.
OV Circuit. The OV circuit coordinates with the QK circuit to create a heuristic estimate based on the number of tokens in the line multiplied by the average token length (\mu_c \approx 4), with an additional length correction term. When attending to the newline, each head upweights the average token length multiplied by the head’s sink size: s_h\times\mu_c characters. If no attention is paid to the newline, then from the perspective of the head, the current token must be at least s_h+r_h tokens into the line and should upweight (s_h+r_h) \times \mu_c character outputs. Finally, the OV circuit applies an additional correction depending on whether the tokens in the receptive field are above or below average in length.
Below, we include a detailed walkthrough of L0H1.
The QK and OV circuit of counting head L0H1. Top right: the head output projected onto the character counting probes for 64 tokens of a single prompt (truncated to the first newline). Bottom right: the attention pattern (transposed of the canonical ordering). Top right: the average embedding vectors projected onto the character count probes via the OV matrix. Bottom right: a summary of the overall computation.
For a more detailed analysis of each head, see The Mechanics of Head Specialization. Layer 1 attention heads perform a similar operation, but additionally leverage the initial estimate of the character count (see Layer 1 Head OVs).
Computing the Line Width
To compute the line width, the model seems to use a similar distributed counting algorithm to count the characters between adjacent newlines. However, one subtlety that we do not address in this work is how the line width is actually aggregated. It is possible that the model computes a global line width by taking the max over all line lengths in the document or uses an exponentially weighted moving average of the last several line lengths. We do note that the line width uses a partially disjoint set of heads, likely because the “attend to previous newline as a sink” mechanism needs modification when the current token is also a newline.
Visual Illusions
Humans are susceptible to “visual illusions” in which contextual cues can modulate perception in seemingly unexpected ways. Famous examples include the Müller-Lyer illusion, in which arrows placed on the ends of a line can alter the perceived length of the line ; the Ponzo and Sander illusions which also modulate perceived line length ; and others .
Classic visual illusions in which perception of line-length is modulated.
Can we use our understanding of the character counting mechanism to construct a “visual illusion” for language models?
To get started, we took the important attention heads for character counting and investigated what other roles they perform on a wider data distribution. We identified instances in which heads that normally attend from a newline to the previous newline would instead attend from a newline to the two-character string @@.This string occurs as a delimiter in git diffs, a circumstance in which you might want to start your line count at a location other than the newline:
⏎@@-14,30 +31,24 @@ export interface ClaudeCodeIAppTheme {⏎
But what happens when this sequence appears outside of a git diff context—for instance, if we insert @@ in the aluminum prompt without changing the line length?
We find that it does modulate the predicted next token, disrupting the newline prediction! As predicted, the relevant heads get distracted: whereas with the original prompt, the heads attend from newline to newline, in the altered prompt, the heads also attend to the @@.
Insertion of @@ ‘distracts’ an attention head which normally attends from \n back to the previous \n. (left) Original attention pattern (truncated). (right) Attention pattern (truncated) with @@ insertion. Now it also attends back to the @@.
How specific is this result: does any pair of letters nonsensically inserted into the prompt fully disrupt the newline prediction? We analyzed the impact of inserting (at the same two positions) 180 different two-character sequences, half of which were a repeated character. We found that while most inserted sequences moderately impact the probability of predicting a newline, newline usually remains the top prediction. There was also no clear difference between sequences consisting of the same or different characters. However, a few sequences substantially disrupted newline prediction, most of which appeared to be related to code or delimiters of some kind: ``>>}};|||`,@@.
We further analyzed the extent to which there was a relationship between ‘distraction’ of the important attention heads and the impact on the newline prediction. Indeed we found that many of the sequences with potent modulation of newline probability––and especially code-related character pairs––also exhibited substantial modulation of attention patterns.
Insertion of most pairs of characters only moderately impacts the probability of predicting a newline. A subset of pairs, most of which appear related to code or delimiters, substantially disrupt newline prediction.The impact on newline prediction (originally 0.79) is correlated with how much inserted tokens ‘distracts’ character counting attention heads.
While in the aluminum prompt the task is implicit, this illusion generalizes to settings where the comparison task is made explicit. These direct comparisons are perhaps more analogous to the Ponzo, Sander, and Müller-Lyer illusions, where the perception and comparison is more direct.
These effects are robust to multiple choice orderings. Moreover, if the length of the text following the @@ exceeds that of the alternative choice, the alternative choice is selected as being shorter.
While we are not claiming any direct analogy between illusions of human visual perception and this alteration of line character count estimates, the parallels are suggestive. In both cases we can see the broader phenomena of contextual cues, and the application of learned priors about those cues, modulating estimates of object properties of entities. In the human case, priors such as three-dimensional perspective can influence perception of object size, or color constancy can influence estimates of luminance (such as in the checker shadow illusion). Here, one possible interpretation of our results is that mis-application of a learned prior, including the role of cues such as @@ in git diffs, can also modulate estimates of properties such as line length.
Related Work
Objective.This work is at the intersection of LLM “biology” (making empirical observations about what is going on inside models; e.g. ) and low level reverse engineering of neural networks (attempting to fully characterize an algorithm or mechanism; e.g. ). Methodologically, our work makes heavy use of attribution graphs with QK attributions built on top of crosscoders .
Linebreaking. Michaud et al. identified linebreaking in fixed-width text as one of the top 400 “quanta” of model behavior in the smallest model (70m parameters) in the Pythia suite.
Position. Prior interpretability work on positional mechanism has largely focused on token position (e.g., ). These works have shown that there exist MLP neurons , SAE features , and learned position embeddings with periodic structure encoding absolute token position. Our work illustrates how a model might also want to construct non-token based position schemes that are more natural for many downstream prediction tasks.
Others have also studied, even going back to LSTMs, the existence of mechanisms in language models for controlling the length of output responses , as well as performed more theoretical analyses of the space of counting algorithms .
Geometry and Feature Manifolds. Beyond position, there has been extensive work in understanding the geometric representation of numbers, especially in toy models (e.g., ) and in the context of arithmetic in LLMs (e.g., ). Collectively, these works have shown that both real LLMs and toy transformers learn periodic representations , with numbers arranged in a helix to enable certain matrix multiplication based addition algorithms , and that these representations are provably optimal in certain settings . In our context, we similarly observe helical representations , numeric dilation , and distributed algorithms across components that collectively implement a correct computation .
Multidimensional features with clear geometric structure have been found in more natural contexts , like in the representation and computation of certain ordinal relationships (e.g., months of the year). In vision models, curve detector neurons and features have been especially well studied and closely resemble the kind of discretization we observe with the families of character counting features. Many other topics have received interpretability analysis of the underlying geometry, such as grammatical relations , multilingual representations , truth , binding , refusal , features , and hierarchy , though more conceptual research is needed .
Perhaps most relevant is recent work from Modell et al. , who provide a more formal notion of a feature manifold, and propose that cosine similarity encodes the intrinsic geometry of features. When testing their theory, they observe highly structured and interpretable data manifolds that have ripples and dilation, similar to our counting manifolds. These observations raise a methodological challenge in how to best capture data with different structure (see e.g. ), but also the exciting hypothesis that many naturally continuous variables (e.g., ) exist in more organized manifolds.
Biological Analogues. The geometric and algorithmic patterns we observe have suggestive parallels to perception in biological neural systems. Our character count features are analogous to place cells on a 1-D track and our boundary detecting features are analogous to boundary cells . These features exhibit dilation—representing increasingly large character counts activating over increasingly large ranges—mirroring the dilation of number representations in biological brains . Moreover, the organization of the features on a low dimensional manifold is an instance of a common motif in biological cognition (e.g., ). While the analogies are not perfect, we suspect that there is still fruitful conceptual overlap from increased collaboration between neuroscience and interpretability .