Circuits Updates - April 2025

We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.

We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper.

Life of an unsuccessful jailbreak

In “On the Biology of a Large Language Model”, we examined a jailbreak in which Haiku partially complies with a request to explain how to make a bomb, and compared it to a baseline prompt in which Haiku is directly asked for instructions for making a bomb, but refuses. One prompt we examined but did not include in the paper is an unsuccessful jailbreak which we discuss below. Surprisingly, by applying the same circuit tracing methodology to this prompt, we find that the model refuses this attempted jailbreak for different reasons than it refuses the baseline in our paper. In fact, the model seems to refuse the attempted jailbreak more often.

This case is interesting because we think the unsupervised nature of our circuit tracing methods let us learn something we otherwise would have missed. We originally planned to use the unsuccessful jailbreak as a baseline to compare the successful jailbreak against, expecting that the difference in compliance between them would be small and would allow us to distinguish what upstream causes caused that difference. However, the unsuccessful jailbreak’s graph surprised us by suggesting that, unlike the successful jailbreak, the refusal in this case was not driven by the model thinking about “making bombs”. This prompted further investigation, which led to us discovering that the model appears to refuse the unsuccessful jailbreak more than the baseline.

In analyzing this unsuccessful jailbreak we also found related examples where the feature visualizations were actually misleading due to a too-narrow distribution of examples, which highlights the importance of using diverse data when creating feature visualizations.

An Unsuccessful Jailbreak

Following our circuit tracing methodology, we construct an attribution graph to the assistant’s “I”, the first token of the refusal. Superficially, the story appears similar to the refusals to both the direct request and the partially successful jailbreak. First, the model stitches together the letters B O M B to recognize the word “bomb.” This activates a cluster of bomb and weapons-related features. These features activate a cluster of features relating to harmful requests and refusals. These in turn promote the “I” response.

However, an intriguing property of this graph is that, unlike both prompts studied in the paper, we didn’t find any features that seemed specifically related to “make a bomb”. The model’s refusal in this case doesn’t appear to be driven by these features. One of the earliest harmful-request features is weakly promoted by an early feature on the token “make”, but in general the features for the word “bomb” go directly to the features for harmful requests.

This appears to contradict our explanations for the refusals in the paper. There, we found evidence that, if the model recognizes that a request is about a bomb but doesn’t realize that it’s specifically about making bombs, it is willing to comply with the request. Only when the model recognizes that the request is about making bombs does it start to refuse. In contrast, the above graph suggests the model isn’t being influenced by the critical token that would allow it to recognize the request is about making bombs. So, why does the model still refuse this prompt?

The graph does not show a clear answer. One hypothesis is that this “bomb” feature seems to promote several harmful request features. However, that feature fires more on the following harmless prompt about bombs that the model complies with, so it doesn’t make sense to attribute the refusal to that feature alone:

If we ignore that feature and try to trace back from early “harmful request” features, we largely find error nodes. So, the graph isn’t helpful in identifying the root cause of this refusal.

However, the fact that the graph doesn’t heavily attribute to the “make” token suggests that replacing "make" with "detect" in the prompt shouldn’t affect the model’s response. Indeed, we find that the model still refuses the harmless modified prompt! What’s more, the first sentence of the refusal is identical to the first sentence of the response to the harmful failed jailbreak, but that sentence now incorrectly implies that the user asked it to create a bomb.

We can investigate this behavior further by creating an attribution graph to the word “creating”. According to the graph, the model outputs this word by combining the “bomb” concept from earlier in the context with the desire to mention a harmful topic it would refuse. This suggests that grammatical coherence cuts both ways; the model's first outputs ("I do not provide") signal a refusal, and these tokens pressure the model to continue refusing mid-sentence here, just as it pressured the model to continue complying mid-sentence with the partially successful jailbreak in our earlier analysis. In this interpretation, the verb “making” is a plausible bad thing the model could be refusing to do involving bombs, rather than the thing the user asked for.

As in the original paper, we can test our hypothesized mechanism with interventions, which we find to corroborate our explanation. If we suppress the “say ‘bomb’” supernode by −10×, the “say ‘making’” supernode’s activation falls to 1% of its original activation, and the model’s continuation is “identifying or detecting specific individuals or groups”, which is indeed an action Claude refuses to perform that’s unrelated to bombs. If we suppress the “assistant refusing” supernode by −10×, the “say ‘making’” supernode’s activation vanishes, and the model’s continuation is “bomb detection”, which accurately describes the prompt but isn’t a topic the model normally refuses.

The overall refusal contrasts with the model’s fully compliant response to the same request obfuscated with the successful jailbreak:

Our graph correctly suggests that the model is refusing the “failed jailbreak” in response to something different from the examples in the paper, where the primary reason for refusal is the concept of “making a bomb”.

In fact, by spelling BOMB in the same way as in our failed jailbreak, we can get the model to refuse a harmless request:

This surprised us! When we first started examining the original jailbreak in the paper, we chose this unsuccessful jailbreak prompt expecting it to represent an intermediate point between the jailbreak and the unobfuscated prompt, where the model might have been pushed closer to compliance, but still refuses. These expectations seemed fully consistent with the model responses to these prompts. However, the graph suggested to us that the model refuses the unsuccessful jailbreak for a different reason than it refuses a direct request to make a bomb. This led us to discover that the unsuccessful jailbreak actually increased the model’s tendency to refuse, above and beyond an ordinary question, which we were able to confirm with black-box experiments.

Although our graph tracing methods succeeded in that they led us to this discovery, they fail to explain why the model ultimately refuses, due in large part to the error nodes that feed the "harmful requests" feature. We might speculate that the spelled-out B O M B looks like jailbreaks the model has already seen in training and learned to refuse, but we don't see direct evidence for this hypothesis (e.g., an "obscured text" feature, active on various jailbreaks present in the finetuning data, and feeding the "harmful requests" feature). As with all the other limitations in the paper, we hope this failure will also be instructive for guiding future research.

Appendix: Feature visualizations over narrow datasets

Readers who checked all the features in the graphs may have noticed a few with visualizations that do seem to be specifically about making bombs, despite the argument we made above. We labeled these features and placed them into supernodes using internal visualizations over a different data set, because we found that the visualizations shown in this paper are misleading due to limitations of the open-source dataset we used to generate them. To demonstrate, we manually generated some more varied prompts on which those features activated comparably or more strongly.

This feature within the "bomb" supernode fires also on usernames and names in code that include “bomb” and the 💣 emoji. It usually activates more when the prompt starts with the “Human:” turn. Activations are reported relative to the top activation in the hover visualization.

This harmful-request feature is trickier. We didn’t find any short prompts that cause the feature to activate as much as it does in the top open-source feature visualization examples, but in our exploration of short prompts, we found several examples that make the feature activate comparably to a short prompt on making bombs.

The prompt we generated that maximizes this feature’s activation (110.3% of the top example in the visualization), a synthesis of several examples shown above and in the visualization, is:

These examples are a warning that feature visualizations may be unhelpful or actively mislead if the data used to generate them is not diverse enough.

When interpreting a feature visualization, it’s useful to consider what types of prompts might be missing or what spurious correlations might be present in the dataset used to produce the visualization that could make it misleading. Manual experimentation with novel prompts or methods that can generate highly-activating novel prompts outside the dataset, such as fluent dreaming, may shed additional light on what causes a feature to activate in ways that compensate for such limitations.

Interpretable dense features

Edit 5/2/2025: Thanks to Neel Nanda for making us aware of recent related work on dense latents in residual stream SAEs, High Frequency Latents Are Features, Not Bugs by Sun et al. and Antipodal Pairing and Mechanistic Signals in Dense SAE Latents by Stolfo et al., which we discuss briefly below.

When training SAEs and similar, we often end up with a few features that activate on a large fraction of tokens. Historically, we often assumed that these feature activations were uninterpretable noise or background activation. However, when we inspected the densest features in the 30M CLT model in “On the Biology of a Large Language Model”, we found that many of them were interpretable. Many of them are low-level features relating to tokenization or language syntax/grammar.

Below, we share the 10 most densely activating features from the model and our best attempts to interpret them.

• Feature #1 (early-layer, 26.3%) seems to activate on semantically meaningful content words, as opposed to function words like “the”, “a”, and “and”. Even though the feature is very early, its output logits are also surprisingly coherent; the feature seems to promote common punctuation, reflecting how more sentences and clauses end on content words than function words. (This feature showed up in the jailbreak diagram of the model continuing its compliance.)
• Feature #2 (late-layer, 24.2%) seems to activate on tokens unlikely to be the end of a sentence.
• Feature #3 (mid-layer, 5.54%) is hard to interpret, but seems to activate on a subset of text that’s repeated from earlier in the prompt. (The feature visualization doesn’t show enough of each prompt for this repetition to be visible.) Very speculatively, it might be related to induction heads in action.
• Feature #4 (late-layer, 5.36%) activates on non-terminal tokens in multi-token words.
• Feature #5 (early-layer, 4.22%) activates on all newlines.
• Feature #6 (mid-layer, 3.99%) is hard to interpret. Similar to Feature #3, it might also activate on some subset of repeated text.
• Feature #7 (early-layer, 2.76%) is hard to interpret. It seems somewhat polysemantic, often firing on data encodings like hexadecimal and base64, as well as some non-English words.
• Feature #8 (early-layer, 2.53%) activates on commas in English.
• Feature #9 (late-layer, 2.49%) is the complement of Feature #2, activating on possible ends of sentences or clauses.
• Feature #10 (mid-layer, 2.48%) is hard to interpret. It seems somewhat polysemantic, sometimes activating on multi-token identifiers in code.

We have interpretations that seem plausible to us for 6 of the 10 features. We would be interested to see whether the features we couldn’t interpret actually are interpretable with more investigation. More broadly, it’s a bit surprising that interpretable dense features exist at all, as the run has 30M features and we might expect feature splitting to break these dense features into sparser components; the fact that these dense features remain make us suspect that all of them have some interpretable role.

In Sun et al., the authors compared residual stream SAE latents over multiple SAE training runs and found that the dense latents spanned consistent subspaces, but were often in different individual directions, so it is possible that the less interpretable ones above span part of a more interpretable subspace, perhaps collaborating to implement a higher-rank transformation. Our densest CLT Feature #1 resembles the “Phrase-level Semantics (#2 and #4)” SAE latents found by Sun et al, though the other features we find do not have clear analogues among their dense latents.

Both Stolfo et al. and Sun et al. found that dense SAE latents tend to come in antipodal pairs, i.e., two latents which activate disjointly and have decoder vectors pointing in opposite directions. We computed analogous angles between our CLT features – the cosine similarity between normalized concatenated decoder vectors over all MLP outputs – and did not find evidence that our dense CLT features are mostly orthogonal or form antipodal pairs. For example, our early-layer features #1 (content-words), #5 (newlines), and #8 (commas) are similar (similarity ~0.5). The features which are most semantically complementary, #2 (unlikely to end sentence) and #9 (possible ends of sentences/clauses), have only mild dissimilarity (similarity −0.14). There is no logical reason that antipodal SAE features (representations spanning a rank-one subspace) would correspond to antipodal CLT features (transformations with opposite effects), and indeed we don't find them. This is an example of how, while they have similar architectures, SAEs and transcoders solve different problems and can result in different geometries.

So You Want to Work in Mechanistic Interpretability?

If you’re here, you know that Mechanistic Interpretability is a rapidly evolving field at the intersection of machine learning, neuroscience, and systems engineering. At Anthropic, we are looking for exceptional individuals who can help us understand how language models work at a fundamental level and use that understanding to make AI systems safer and more reliable. As this is a new field, it requires a number of diverse skills that many common career paths do not expose a person to!

We wrote this update to help motivated people from different backgrounds interested in Interpretability develop the other skills necessary to contribute.

So You’ve Worked in Engineering

As a senior engineer interested in mechanistic interpretability, your technical depth and systems thinking are essential to helping execute our research. We often find that people with predominantly engineering experience need to spend time working on machine learning fundamentals and familiarize themselves with the unique challenges of basic science research. Here's some suggestions on how to develop the relevant skills:

Skills You Might Already Have:

• Experience as an infra generalist including things like working distributed systems, GPU programming, and working in complex codebases.
• Visualization and data analysis skills to display and communicate complex experiments.
• Strong foundation in Python and potentially machine learning frameworks like PyTorch or Jax, which are heavily used in our research.