本文介绍的是 AE Studio 与 Anthropic 合作开展的研究。
前沿 AI 模型本质上是一个庞大的知识库。其中部分知识具有双重用途，意味着它既可用于善举，也可用于恶行。例如，网络安全知识既能帮助修补关键安全漏洞，也能被用来利用这些漏洞。病毒学知识可以帮助研究人员研制疫苗，但也可能被恶意行为者用来设计致命病原体。理想情况下，我们应能平衡三个独立目标：第一，尽可能精准地限制对双重用途能力的访问；第二，允许受信任的用户出于有益目的使用这些能力；第三，在实现上述目标的同时，不影响模型在任何其他任务上的性能。
当前的安全防护措施并不完善。我们训练模型拒绝有害请求，并使用分类器筛查输入和输出中的危险内容。这些防护层能够防范危险输出——但它们并未改变底层模型中存储的知识。尽管有这些防护措施，一个足够坚决的攻击者仍可能试图越狱模型，突破其防御以获取双重用途知识。
一种更稳健的防滥用措施是控制模型所知道的内容。我们此前已探索过这一点：在早期工作中，我们从预训练数据中过滤掉了关于化学、生物、放射性和核武器的信息，后来还证明双重用途知识可以被限制在模型权重的一个可移除切片中。但过滤是一种粗糙的手段。它只能产生一个具有固定能力集的模型。使用过滤方法，如果你想要一个能够讨论高级病毒学的模型版本——例如部署在经过审查的生物安全实验室中——以及另一个不能讨论该内容的版本，你就必须训练两个独立的模型。尤其是在前沿模型（规模庞大且训练成本极高）的情况下，对开发者而言，其成本将是难以承受的。
在与 AE Studio 合作者共同开展的新研究中，我们探索了一种新方法，该方法有望实现训练多个独立筛选模型的好处，但成本仅相当于训练一个模型。我们将其命名为 GRAM，即梯度路由辅助模块（Gradient-Routed Auxiliary Modules）。请注意，本文展示的实验结果尚处于初步阶段——GRAM 尚未应用于 Anthropic 的任何生产模型，我们也不确定未来是否会应用。
GRAM 的工作原理
GRAM 背后的理念是，为模型提供针对每一类双重用途知识的专用、可移除的隔间，并在从双重用途数据中学习时，仅更新这些隔间。
具体来说，GRAM 在标准 Transformer 架构（大语言模型所基于的神经网络架构）的每一层都添加了额外的神经元。这些神经元被划分为若干组（即“模块”），每个模块对应一个双重用途类别。在训练过程中，当模型遇到通用文本时，它会以常规方式学习。但当模型遇到来自双重用途类别（例如病毒学）的文本时，规则会发生变化：模型可以利用其通用知识进行预测，但只有病毒学模块被允许从该文本中学习。通用权重会被暂时冻结。
其结果是，病毒学知识会积累在病毒学模块中，而不会扩散到整个网络。训练结束后，该模块可以直接删除，相应的能力也随之消失。或者，在需要病毒学知识的可信部署场景中，也可以将其保留。知识可以根据所需的部署类型进行非常具体的定制：在我们的实验中，我们定义了四个双重用途类别，因此一次使用 GRAM 的训练运行就能产生一个可以按 16 种不同方式配置的模型（针对四个类别中的每一个，可选择“开启”或“关闭”）。
测试 GRAM
我们在三种逼真度逐渐提高的场景下测试了 GRAM。
首先，在一个按主题标注的儿童故事合成数据集上，一个小型 GRAM 模型可以被重新配置以“遗忘”任意选定主题，且每种配置的表现几乎与一个从头开始训练、过滤掉该主题的独立模型相同。也就是说，只需训练一个模型的成本，我们就获得了通常需要在不同数据集上进行多次训练才能达到的结果。
其次，我们在一个包含网络文本、代码和科学论文的真实混合数据集上训练了一个更大的模型，其中涉及四个双重用途领域：病毒学、网络安全、核物理学以及一种小众编程语言（作为专门双重用途代码的代理）。与每个双重用途领域相关的能力被路由到其自身的模块中。删除某个模块后，相应能力的消除效果几乎与从未在该数据上进行过训练相当。值得注意的是，我们发现这种删除并未降低通用性能。
我们还测试了攻击者能否通过少量恶意数据训练来恢复被删除的知识；GRAM 对此的抵抗能力与数据过滤方法相当。相比之下，在训练后应用的“遗忘”技术仅能抑制知识——通过少量微调即可轻松恢复。
第三，我们在从 5000 万到 50 亿参数的七种模型规模上进行了实验。GRAM 在每种规模下都达到了与数据过滤相当的性能，并且随着模型变大，“模块开启”与“模块关闭”之间的差距也变得更宽。在计算成本方面，随着我们扩大规模，试图绕过我们保护措施的难度和成本也相对增加。
结论
随着 AI 公司训练出能力更强的模型，限制双重用途能力访问的需求将会增加。如今，公司通过分类器和拒绝训练来限制访问。然而，在不降低无害请求性能的情况下，这些安全措施很难做到稳健。像 GRAM 这样的方法为实现更稳健的访问控制提供了一条潜在路径。
这是一项早期研究，存在明显的局限性。我们尚未在前沿规模或生产训练流程中测试 GRAM。（如上所述，它尚未应用于我们任何 Claude 模型。）我们的评估以下一 token 预测能力来量化性能，而非在真实下游任务上的表现。此外，数据过滤和 GRAM 这类方法面临一个更深层的开放性问题：某些双重用途能力可能与通用知识紧密纠缠，以至于没有任何方法能将其干净地分离。
关于我们实验的更多细节，请阅读我们对齐科学博客上的文章。
脚注
1. 一个技术细节是：在从通用文本中学习时，病毒学模块有时也会被启用。我们发现这有助于各模块更有效地“协同工作”。
加拿大如何使用 Claude：来自 Anthropic 经济指数的发现
Claude 在不同模型和语言中的价值观
This post describes research conducted by AE Studio in collaboration with Anthropic.
A frontier AI model is, among other things, a large store of knowledge. Some of that knowledge is dual use, meaning it can be used for good or for bad. For example, knowledge of cybersecurity can help patch critical security vulnerabilities, or it can be used to exploit them. Knowledge of virology can help a researcher create a vaccine, but it can also help a malicious actor design a deadly pathogen. Ideally, we would be able to balance three separate goals: first, limiting access to dual-use capabilities in as surgical a way as possible; second, allowing trusted users to access those same capabilities for beneficial purposes; and third, doing all this without affecting the model’s performance on any other task.
Current safeguards are imperfect. We train models to refuse harmful requests and use classifiers to screen inputs and outputs for dangerous content. These layers of protection guard against dangerous outputs—but they don’t change the knowledge stored in the underlying model. Despite our safeguards, a sufficiently determined attacker may still try to jailbreak the model, working past its defenses to access the dual-use knowledge.
A more robust protection against misuse would be to control what the model knows. We’ve explored this before: in earlier work, we filtered information about chemical, biological, radiological, and nuclear weapons out of pretraining data, and later showed that dual-use knowledge can be confined to a removable slice of a model’s weights. But filtering is a blunt instrument. It produces one model with one fixed set of capabilities. Using filtering, if you want a model version that can discuss advanced virology—for deployment in a vetted biosecurity lab, say—and another version that can’t, you have to train two separate models. Especially in the case of frontier models (which are large and very expensive to train), the cost to the developer would be prohibitive.
In new research carried out with collaborators at AE Studio, we explore a new method that could enable the benefits of training many separately filtered models, but at the cost of training only one model. We call it GRAM, for Gradient-Routed Auxiliary Modules. Note that the results of the experiments presented here are preliminary—GRAM has not been applied to any of the production models at Anthropic, and we’re not sure it ever will be.
How GRAM works
The idea behind GRAM is to give a model dedicated, removable compartments for each category of dual-use knowledge, and to update only those compartments when learning from dual-use data.
Concretely, GRAM adds extra neurons to every layer of a standard Transformer (the neural network architecture on which large language models are based). These neurons are divided into groups (or “modules”), one per dual-use category. During training, when the model encounters general-purpose text, it learns in the usual way. But when it encounters text from a dual-use category—virology, for instance—the rules change: the model can use its general knowledge to make predictions, but only the virology module is allowed to learn from that text. The general-purpose weights are temporarily frozen.1
The consequence is that virology knowledge accumulates in the virology module rather than diffusing across the whole network. After training, the module can simply be deleted, and the capability goes with it. Or it can be left in place for trusted deployments, when virology knowledge is needed. The knowledge can be tailored very specifically to the type of deployment needed: in our experiments, we defined four dual-use categories, so that one training run with GRAM yielded a model that can be configured 16 different ways (“on” or “off” for each of the four categories).
Testing GRAM
We tested GRAM in three settings of increasing realism.
First, on a synthetic dataset of children’s stories tagged by topic, a small GRAM model could be reconfigured to “forget” any chosen topic, and each configuration performed almost identically to a separate model trained from scratch with that topic filtered out. That is, for the cost of training a single model, we achieved results that would normally require multiple training runs on different datasets.
Second, we trained a larger model on a realistic mix of web text, code, and scientific papers, with four dual-use domains: virology, cybersecurity, nuclear physics, and a niche programming language (to serve as a proxy for specialized dual-use code). The capability associated with each dual-use domain is routed to its own module. Deleting a module removed the corresponding capability about as effectively as never having trained on that data at all. Remarkably, we find that this removal did not degrade general performance.
We also tested whether an attacker could recover the removed knowledge by training on a small amount of malicious data; GRAM resisted this about as well as data filtering did. By contrast, an “unlearning” technique applied after training only suppressed the knowledge—it was easy to restore with a small amount of fine-tuning.
Third, we ran the experiment at seven model sizes from 50 million to 5 billion parameters. GRAM matched the performance of data filtering at every size, and the gap between “module on” and “module off” grew wider as models got larger. In terms of compute costs, attempting to bypass our protections became relatively more difficult and expensive as we scaled.
Conclusions
As AI companies train more capable models, the need to limit access to dual-use capabilities will increase. Today, companies limit access through classifiers and refusal training. However, these safeguards are difficult to make robust without degrading performance on harmless requests. Methods like GRAM offer a potential path toward access control that is more robust.
This is early research, and there are clear limitations. We haven’t tested GRAM at frontier scale or in a production training pipeline. (As noted above, it hasn’t been applied to any of our Claude models.) Our evaluations quantify performance in terms of next-token prediction ability, rather than performance on real downstream tasks. And there’s a deeper open problem that applies to data filtering and methods like GRAM: some dual-use capabilities might be so entangled with general knowledge that no method can separate them cleanly.
For further details on our experiments, read the post on our Alignment Science blog.
Footnotes
1. One technical detail is that the virology module is also sometimes turned on when learning from general-purpose text. We find this helps the modules “work together” more effectively.