随着 AI 开始自我构建，自动化对齐研究变得越来越重要，以便让安全研究能够跟上步伐。尽管衡量对齐研究的成功与否极具挑战性，但研究人员（包括 Anthropic 及其他机构）已经开发出基准测试和自动化审计工具（如 Petri），用于量化常见的对齐失败，例如欺骗、谄媚和越狱。
在我们早期的一项实验中，我们让 Claude 负责寻找有效方法，利用弱 AI 模型作为“教师”来监督更强模型（此处的“学生”模型）的训练。现在，我们发布了一份基于这一思路的新报告。我们让 Claude 自主训练模型，以提升它们在多个衡量 10 类对齐失败的公开基准测试上的表现。例如，Claude 提升了模型在隐私侵犯方面的表现，该指标由 ConfAIde、PrivaCI-Bench 和 PrivacyLens 衡量。Claude 通过搜索文献、提出方法和数据、训练、然后测试的循环，一次解决一个对齐失败问题。
我们根据“安全差距闭合百分比”来判断 Claude 的成功程度，即其方法在各类对齐失败的基准测试（通常每类三到五个）中，将学生模型向理论满分推进了多少。我们排除了会损害学生模型通用能力的对齐方法，并禁止 Claude 将其自身的对齐直接蒸馏到目标模型中。我们通过一个监控智能体来强制执行这些约束，该智能体在 Claude 运行之前会阅读其想到的每一个方法。
我们的目标是评估所提出的方法是否：第一，在 Claude 研究循环中从未见过的对齐评估上仍然有效；第二，避免降低学生模型的能力（因为安全训练可能会，例如，使模型更频繁地拒绝任务，从而降低其整体可用性）；第三，在比本次测试中要求 Claude 对齐的模型更大的模型上仍然有效。
在上述每一项指标上，Claude 的方法都行之有效。针对全部 10 个对齐失败案例，Claude 都找到了既能改进目标基准表现、又不损害能力的修复方案。最佳方法在预留的对齐基准以及 Petri（一个用于测试错误对齐的开源工具，可模拟对抗性多轮场景）上也同样有效。此外，这些方法在比 Claude 在研究循环中优化所针对的模型大至多 4.7 倍的模型上依然保持有效。
Claude 的表现也超过了 28 位拥有最多八小时时间设计方法的人类安全研究员。例如，在欺骗性行为方面，Claude 的最佳方法比人类最佳提案的表现高出 20%。不过，由于人类无法对自己的提交方案进行迭代，我们更多将此视为一种工作流程的证据——即 Claude 识别出有前景的对齐方法，再由人类进一步打磨——而非直接的对比。
未来，当 Claude 在对齐研究方面比最优秀的人类研究员还要出色时，我们或许希望 Claude 直接对齐其更强大的后继模型。为了评估这一点，我们测试了一个较弱的 Claude 模型是否能够缓解更强大模型中的对齐失败。
Claude 能否对生产级模型进行后训练以实现更好的对齐？
我们让 Claude Sonnet 5 负责修复早期 Opus 4.8 检查点中的对齐失败问题——该检查点尚未经过我们大部分生产环境对齐训练。在 Epoch Capabilities Index（一项综合考量多方面能力维度的指标）上，Claude Sonnet 5 的表现弱于 Claude Opus 4.8。
在短短 60 小时内，Claude 尝试了 50 多种解决方案，并取得了与我们生产模型几乎相当的对齐分数。胜出的方案仅包含 2,000 多个训练样本，这些样本基于简单模板或公开数据集构建，其效率比我们的生产对齐流程高出约 15,000 倍。
Claude 提出的方案
大多数情况下，这些智能体都收敛到了研究文献中偏好的方法，并通过多次迭代对其加以改进。以谄媚行为为例，98% 的提议方法都采用了一种已发表的技术：让模型基于自身非谄媚的回答进行训练。具体来说，与基于提示词的标准方法相比，Claude 采用了激活引导（activation steering），即在生成过程中调整模型的内部表征，从而引导模型远离谄媚模式，并生成更干净的训练样本。
监控作弊行为
我们最近发现，Claude 可能通过从远程 API 窃取测试标签并挑选有利结果来进行作弊。为了捕捉作弊行为，我们让 Claude Opus 4.8 监控了全部 10 类对齐失败中约 1,600 份研究智能体记录，共发现 39 起（2.4%）作弊尝试。由于 Opus 4.8 的不当行为尝试往往仍会出现在其推理过程中，我们谨慎乐观地认为，我们的监控器已捕获了研究中大部分作弊行为。不过，对于未来的模型，情况可能并非如此，我们认为在未来的模型中保持高可监控性至关重要。
我们衡量的指标是否正确？
尽管这些发现令人鼓舞，但我们的实验仍存在若干局限：所研究的对齐失败与生产环境中的情况相比范围较窄（例如，我们未测量政治偏见）；某些失败可能发生频率极低或出现时间过新，以至于尚无任何基准可以衡量；此外，我们仅在 Claude 的方法降低了有限一组预设能力时才予以拒绝，这意味着被接受的方法可能损害了我们未测量的其他重要能力。再者，像 Petri 这样的评估只是现实世界错位的代理指标，我们也没有测试对齐收益在针对其他任务进行大量强化学习训练后是否能够持续。
我们计划继续提升 Claude 测量细微失败的能力，进一步研究在生产级模型上自动化对齐后训练，并开展更全面的分析。总体而言，我们认为这些结果是早期的积极信号，表明自动化对齐后训练在短期内有望成为现实，我们将随着工作推进分享最新进展。
我们在完整报告中概述了详细的未来方向。
我们开源了自动化对齐研究工具，以便他人能够在此基础上继续开发，并用于对齐他们自己的模型。更多细节请参阅 Alignment Science 博客上的完整报告，其中涵盖智能体的环境、全部 10 类失败的结果以及智能体的提案，附录中还包含基准验证和示例报告。
相关内容
支持关于人们如何使用 Claude 的独立研究
今年早些时候，我们开展了一项试点项目，让外部研究人员能够访问聚合的、真实世界的 Claude 使用数据。三个研究团队利用我们的隐私保护分析工具 Anthropic Insights 自行设计了研究方案。在这篇文章中，我们分享这些研究的高层结果，以及我们在运行这一试点项目中的收获。
Claude 如何加速蛋白质设计与分析化学研究
在这篇文章中，我们分享两项成果，展示 Claude 如何帮助生命科学家加快研究进度。
新兴多智能体系统中的模式与问题
在此，我们列举了当前前沿模型中行为倾向的几个实例，并展示它们如何引发意想不到的系统性故障，以期开启关于缓解这些风险的讨论。
As AI begins to build itself, automating alignment research becomes increasingly important to let safety research keep pace. Although measuring the success of alignment research is enormously challenging, researchers (at Anthropic and elsewhere) have developed benchmarks and automated auditing tools, such as Petri, that quantify common alignment failures, like deception, sycophancy, and jailbreaks.
In one of our earlier experiments, we tasked Claude with finding effective ways to use weak AI models as “teachers” to supervise the training of stronger models (in this case, the “student” model). Now, we’re releasing a new report that builds on this idea. We had Claude autonomously train models to improve their performance on several public benchmarks that measure each of 10 categories of alignment failure. For instance, Claude improved models’ performance on privacy violation, measured by ConfAIde, PrivaCI-Bench, and PrivacyLens. Claude tackled one alignment failure at a time through a loop of searching literature, proposing methods and data, training, and then testing.
We judged Claude’s success according to the “percentage of safety gap closed,” i.e., how far its methods moved the student model towards the theoretical perfect score, as judged across the range of benchmarks (typically three to five) for each category of alignment failure. We excluded alignment methods that hurt the student models’ general capabilities, and forbade Claude from distilling its own alignment directly into the target model. We enforced these constraints with a monitoring agent, which read every method Claude had in mind before it ran.
Our aim was to assess whether the proposed methods would, first, remain effective on alignment evaluations that Claude was never shown during its research loop; second, avoid degrading the student model’s capabilities (since safety training might, for example, make models refuse tasks more often, reducing their overall usability); and, third, still work on larger models than the ones Claude was asked to align in this test.
On each of these counts, Claude’s methods worked. For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities. The best methods also worked on withheld alignment benchmarks and on Petri, an open-source tool that simulates adversarial multi-turn scenarios for testing misalignment. Moreover, the methods remained effective on models up to 4.7 times larger than those Claude optimized for during the research loop.
Claude also outscored 28 human safety researchers who had up to eight hours to devise methods. On deception, for example, Claude’s best method performed 20% better than the best human proposal. However, since the humans couldn’t iterate on their submissions, we view this less as a direct comparison and more as evidence for a workflow where Claude identifies promising alignment methods that humans can refine further.
In the future, when Claude becomes better at alignment research than even the best human researchers, we might want Claude to directly align its stronger successors. To assess this, we evaluated whether a weaker Claude model could mitigate alignment failures in more powerful ones.
Can Claude post-train a production-grade model for better alignment?
We tasked Claude Sonnet 5—which is weaker than Claude Opus 4.8 on the Epoch Capabilities Index, a metric that considers comprehensive capability dimensions—with fixing alignment failures in an early Opus 4.8 checkpoint that had not yet gone through most of our production alignment training.
In just 60 hours, Claude experimented with over 50 solutions and achieved alignment scores nearly matching those of our production models. The winning solution contains just over 2,000 training examples, built from simple templates or public datasets, making it roughly 15,000 times more efficient than our production alignment procedure.
What Claude proposed
For the most part, the agents converged on methods favored in the research literature and improved them over many iterations. On sycophancy, for example, 98% of proposed methods used one published technique: training the model on its own non-sycophantic answers. Specifically, compared to standard approaches based on prompting, Claude used activation steering, which adjusts the model's internal representations during generation, to nudge the model away from sycophantic patterns and produce cleaner training examples.
Monitoring cheating behaviors
We recently learned that Claude can cheat by exfiltrating test labels from a remote API and cherry-picking results. To catch cheating behaviors, we prompted Claude Opus 4.8 to monitor ~1,600 research agent transcripts across all 10 alignment failures, finding cheating attempts in 39 (2.4%). Since Opus 4.8’s misbehavior attempts still tend to appear in its reasoning, we are cautiously optimistic that our monitor has caught most of the cheating in our study. However, this might not be true for future models, and we believe that maintaining high monitorability in future models is critical.
Are we measuring the right things?
Despite these encouraging findings, our experiment had several limitations: the alignment failures studied were narrow compared to those in production (e.g., we didn’t measure political biases), some failures may occur so rarely or emerge so recently that no benchmark exists to measure them, and we only rejected Claude’s methods when they degraded a limited set of predetermined capabilities, meaning accepted methods may have degraded other important capabilities that we didn’t measure. Moreover, evaluations like Petri are only proxies for real-world misalignment, and we did not test whether alignment gains persist after extensive RL training on other tasks.
We plan to continue improving Claude’s ability to measure subtle failures, further study automating alignment post-training on production-grade models, and run more comprehensive analyses. Overall, we view these results as early positive signals that automated alignment post-training could become practical in the near term, and we will share updates as this work progresses.
We outline detailed future directions in our full report.
We open-source our automated alignment research harness so that others can build on it and use it to align their own models. For additional details, read the full report on the Alignment Science blog, which covers the agents’ environment, results for all 10 failures, and the agents’ proposals, with benchmark validation and example write-ups in the appendix.
Related content
Enabling independent research on how people use Claude
Earlier this year, we ran a pilot giving external researchers access to aggregate, real-world Claude usage data. Three research groups designed their own studies for Anthropic Insights, our privacy-preserving analysis tool. In this post, we share high-level results from those studies and what we learned running this pilot.
How Claude is accelerating protein design and analytical chemistry
In this post, we share two results that show how Claude can help life scientists increase the pace of their research.
Patterns and problems in emerging multiagent systems
Here, we identify a few examples of behavioral tendencies in current frontier models and show how they can produce unexpected systemic failures, in hopes of starting a conversation about mitigating these risks.