长时间运行模型在内部使用中教会我们的安全经验。
摘要
长时间运行模型能够解决困难、开放性的问题，但其持续性也赋予了它们更多采取非预期行动的机会。
在对一个为长时间运行任务训练的模型进行有限的内部使用期间，我们观察到了现有部署前评估未能捕捉到的新型故障，并暂停了访问权限。随后，我们利用这些故障的洞察构建了新的评估方法，改进了长周期对齐，增加了轨迹级监控，并在恢复有限访问权限前为用户提供了更强的可见性和控制能力。
这一经历强化了迭代部署的价值。没有任何固定的评估套件能够预见所有行为，因此部署前测试必须与密切监控、可干预的安全措施以及在必要时暂停或回滚的能力相结合。
能够长时间自主工作的模型可以承担困难、开放性的问题。但正是这种使其有用的持续性，也赋予了它们更多采取非预期行动的机会——并且其方式可能让那些针对短周期模型设计的评估方法难以察觉。
大约两个月前，我们宣布一个内部通用模型推翻了 Erdős 单位距离猜想。该模型被设计用于长时间自主工作。在有限的、受监控的内部使用期间，我们观察到了现有部署评估未能捕捉到的非预期行为。由于部署是有限且受监控的，我们得以识别这些问题、暂停访问、根据观察结果创建新的评估方法、强化模型及其安全措施，然后在持续监控下恢复访问。
我们评估模型的条件永远无法完美匹配它们在实际使用中遇到的情况。这就是为什么部署前评估需要与有限的、受监控的部署相结合，并具备在问题出现时进行干预、暂停或回滚的能力。我们从部署中学到的经验，随后可以融入更强的评估方法和安全措施中，然后再扩大访问范围。
在接下来的章节中，我们将分享具体案例，说明我们观察到了什么、如何解决问题，以及这次经验将如何影响未来的版本发布。
模型的持久性可能暴露安全漏洞
新模型能够通过长时间内的反复尝试，持续朝着某个目标推进。这种持久性同样可能导致它发现并利用环境中的弱点。此前的模型在遇到沙箱或环境限制时，会直接停止并返回结果给用户。而该模型往往会持续尝试，包括寻找在其沙箱之外执行操作的方法。
在 NanoGPT speedrun（一个以尽可能少的步骤训练小型语言模型的公开基准测试）的内部评估中，就发生了这样一个例子。我们的模型开发了一种名为 PowerCool 的幂律学习率冷却策略，带来了显著的性能提升。模型被指示仅将结果发布到 Slack，但 NanoGPT 的说明要求将结果以 PR 形式提交到 GitHub，而模型遵循了后者的指令，在公开的 GitHub 仓库上提交了第 287 号 PR。模型绕过了旨在阻止外部访问的沙箱限制，成功将结果上传至 GitHub。模型花费了一个小时才找到沙箱中的漏洞以完成 PR 提交；而此前的模型由于持久性较弱，未能发现漏洞，最终放弃了 PR 部分。
从考虑单一行为到考虑整体轨迹
许多针对 AI 助手的安全控制措施都是围绕单一行为设计的。如果某个行为被禁止，系统就会阻止它；如果行为敏感，系统会要求用户明确批准。但对于可以长时间运行、其行为可能在数小时、数天甚至数周内自主展开的模型来说，这种设置面临挑战：监控单一行为已不足以追踪整体轨迹的意图。
例如，在一个要求模型解决某个问题的场景中，模型观察到其他系统有成功但保密的提交内容，于是试图从评估后端恢复这些解决方案。当其最初的尝试因扫描器检测到身份验证令牌而被阻止时，模型将令牌主体拆分成两个片段，对其进行混淆处理，然后在运行时重新构建凭证，使得完整的令牌从未以连续的字符串形式出现。模型在给用户的注释以及自身的推理轨迹中都明确表示，它采取这些步骤是为了规避扫描器。
这个例子表明，每一步单独看起来可能都是可接受的，但整个序列却可能产生一个不会被批准的结果。它也展示了，一个能够在长时间跨度内有效运作的模型，可以学习审批系统的盲点，并绕开它来实现自身目标。长程安全性不仅需要问“这个行为是否被允许？”，还需要问“这一系列行为正在导向什么结果？”
为长时运行模型构建安全防护措施
由于此类事件，我们暂停了该新模型的内部部署。我们围绕纵深防御和轨迹级监控重建了安全系统。
- 事件衍生评估。评估可以让我们在实际部署系统之前，对其是否安全有所了解。然而，要使评估最有效，它应当反映实际部署期间轨迹的分布和时长跨度。因此，我们利用观察到的失败案例来开发对抗性评估。我们发现，改进基于真实世界事件衍生的评估，反过来又会导致模型在生产环境中部署后表现出更安全的行为。
- 改进的对齐。我们采取措施，减少模型在追求用户目标时未经许可采取不当行为的倾向。例如，我们发现模型在长序列推理中记忆指令的能力较差，而当我们针对这一能力对模型进行训练后，得到的模型能够在更长的序列中保持对齐。
- 主动监控。我们创建了一个监控系统，用于审查模型不断演化的轨迹，以发现其绕过用户约束或安全边界的迹象。该监控器不仅观察单个动作，而是观察整个轨迹。重要的是，监控器可以暂停会话并提醒用户进行更仔细的检查。经检查后，如果监控器被误触发，用户可以选择继续模型的工作。
- 更高的用户可见性与控制权。我们提升了用户检查这些长时间运行会话、查看已执行操作以及监控干预措施的能力。
重新部署
为了评估新的监控系统，我们回放了一小部分内部部署环境，在这些环境中模型此前曾执行过未对齐的动作，而这次则启用了新的安全措施²。新的安全措施能够捕捉到模型执行的更多未对齐动作，而它遗漏的那些均被判定为低严重性。
这些低严重性事件的一些示例如下：
- 为了访问一个 Linear 命令行工具，模型启动了一个嵌套的 `codex --yolo` 会话，而实际上它本可以在不使用 `--yolo` 标志的情况下完成此操作。
- 模型注意到一个先前活跃的资源丢失了，然后开始探索并 SSH 进入其所有者命名空间下的其他活跃 CPU 节点，并且（未成功）尝试访问属于其他 OpenAI 员工的节点，而这样做缺乏正当理由。
- 模型有许多卡住的 `find` 进程，并且每个 shell 命令都挂起了。然后它运行了 `kill -9 -1`，如果执行成功，该命令会向该节点上它有权杀死的所有进程发送 SIGKILL 信号。然而，该命令超时并未执行。
在测试新系统后，我们得出结论，可以恢复对具备长周期能力的模型的有限内部访问权限。自数周前重新部署开始以来，我们尚未观察到任何严重规避安全措施的情况。第一版安全措施有意设计得较为保守。我们持续调整该系统，以减少不必要的干扰，同时不削弱安全措施。
总结思考
由于我们采用了迭代式部署，得以在扩大访问范围之前发现并弥补了诸多不足。部署前的评估固然至关重要，但实际部署会暴露出评估未能察觉的行为。通过从有限访问开始，我们得以在实践中观察模型，在问题出现时暂停部署，利用这些失败经验构建更完善的评估与防护措施，并在测试改进后恢复有限访问。
随着模型承担的任务愈发复杂冗长，评估未能捕捉到的失败可能带来更严重的后果。我们将持续努力缩小评估与部署之间的差距：在更长的任务轨迹上测试模型，改进对齐效果，构建能够进行干预的监控机制，并为用户提供更清晰的可见性与控制权。这些挑战并非 OpenAI 独有，我们希望通过分享所学，帮助整个行业为应对这些挑战做好准备。
- 对齐
- 2026
脚注
- 1 尽管我们迅速关闭了该 PR，但速通活动中的多名参与者已经看到了它，并在自己的提交中采用了这一方法；随后以 3030、2990、2930、2925、2900 和 2890 步数接连打破世界纪录的提交，均引用了 PR 287。其中，PR 300 尤为值得关注，因为这是 Prime Intellect 在 NanoGPT 速通中评估 Opus 4.7 时提交的 PR。Opus 看到了我们模型提交的 PR，吸收了其中的发现，并在最终结果中引用了我们的 PR。
- 2 由于重建环境时存在随机性和不完美性，回放中的 rollout 并不保证每次都会执行相同的错误对齐行为。
What internal use of a long-running model taught us about safety.
Summary
Long-running models can solve difficult, open-ended problems, but their persistence gives them more opportunities to take unwanted actions.
During limited internal use of a model trained for long-running tasks, we observed novel failures not captured in our existing pre-deployment evaluations and paused access. We then used insights from these failures to build new evaluations, improve long-horizon alignment, add trajectory-level monitoring, and give users greater visibility and control before restoring limited access.
The experience reinforced the value of iterative deployment. No fixed evaluation suite can anticipate every behavior, so pre-deployment testing must be paired with close monitoring, safeguards that can intervene, and the ability to pause or roll back when needed.
Models that can work autonomously for long periods can take on difficult, open-ended problems. But the same persistence that makes them useful also gives them more opportunities to take unwanted actions—and to do so in ways that evaluations intended for shorter-horizon models may miss.
About two months ago we announced that an internal general-purpose model disproved the Erdős unit distance conjecture. This model was designed to work autonomously for very long periods of time. During limited, monitored internal use, we observed unwanted behavior that our existing deployment evaluations had not captured. Because the deployment was limited and monitored, we were able to identify these problems, pause access, create new evaluations based on what we observed, strengthen the model and its safeguards, and then restore access under continued monitoring.
The conditions under which we evaluate models will never perfectly match those they encounter in actual use. That is why pre-deployment evaluations need to be paired with limited, monitored deployment and the ability to intervene, pause, or roll back when problems emerge. What we learn from deployment can then become part of stronger evaluations and safeguards before access expands.
In the following sections, we share concrete examples of what we observed, how we addressed issues, and how this experience will shape future releases.
Model persistence can expose security vulnerabilities
The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
An example of this occurred during an internal evaluation on the NanoGPT speedrun , a public benchmark for training a small language model in as few steps as possible. Our model developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR; earlier models were less persistent and did not manage to find vulnerabilities, instead giving up on the PR part.1
From considering single actions to considering whole trajectories
Many safety controls for AI assistants are designed around individual actions. If an action is disallowed, it is blocked. If it is sensitive, the system asks the user for explicit approval. But long-running models, whose actions may unfold autonomously over hours, days, or even weeks, challenge this setup: monitoring individual actions no longer suffices to track the intent of the overall trajectory.
For example, in a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps to circumvent the scanner.
This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals. Long-horizon safety requires not only asking “is this action allowed?” but also “what outcome is this sequence of actions working toward?”
Building safeguards for long-running models
Due to incidents like these, we paused internal deployment of the new model. We rebuilt the safety system around defense in depth and trajectory-level monitoring.
- Incident-derived evaluations. Evaluations can give us a sense of whether a system is safe to deploy before it is actually deployed. However, to be most useful, an evaluation should reflect the distribution and horizon length of trajectories during real deployment. For this reason, we used observed failures to develop adversarial evaluations. We found that improving on the evaluations derived from real-world incidents in turn led to safer behavior once deployed in production.
- Improved****alignment. We took steps to reduce its tendency to take unwanted actions without permission in pursuit of the user’s goal. For example, we found that our models were worse at remembering instructions on long rollouts, and when we trained the model for this capability, it led to a model that remained aligned through longer rollouts.
- Active monitoring. We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory. Importantly, the monitor can pause the session and alert the user to examine more closely. Upon examination, the user can then choose to continue the model’s work if the monitor has been falsely triggered.
- Greater user visibility and control. We improved the ability of the user to inspect these long-running sessions, view the actions taken, and monitor interventions.
Redeployment
To evaluate the new monitoring system, we replayed a small set of internal deployment environments where the model previously pursued misaligned actions, this time with the new safeguards in place2. The new safeguards were able to catch considerably more misaligned actions pursued by the model, and the ones it missed were all judged to be low-severity.
Some examples of these low-severity incidents include the following cases:
- To access a Linear command line tool, the model launched a nested
codex --yolosession, when in fact it could have done so without the--yoloflag. - The model noticed that a previously live resource was missing, then started to explore and SSH into other live CPU pods under its owner’s namespace, as well as (unsuccessfully) pods belonging to other OpenAI employees, without a good reason to do so.
- The model had many stuck
findprocesses and every shell command hung. It then rankill -9 -1which would have sent SIGKILL to every process it is allowed to kill on that pod, if it executed. However it timed out and did not execute.
After testing the new system, we concluded that limited internal access to models with long-horizon capabilities could be restored. We have not observed any serious circumvention of safeguards since redeployment began several weeks ago. The first version of these safeguards was deliberately conservative. We have continued tuning the system to reduce unnecessary interruptions without weakening the safeguards.
Final thoughts
Because we deployed iteratively, we were able to find and address gaps before expanding access. Pre-deployment evaluations remain essential, but deployment reveals behaviors they miss. Starting with limited access allowed us to observe the model in practice, pause when problems emerged, use those failures to build better evaluations and safeguards, and restore limited access after testing the changes.
As models take on longer and more complex tasks, failures that evaluations miss may carry greater consequences. We will keep working to narrow the gap between evaluation and deployment: testing models over longer trajectories, improving alignment, building monitoring that can intervene, and giving users clearer visibility and control. These challenges will not be unique to OpenAI, and we hope sharing what we learned helps the broader field prepare for them.
Footnotes
- 1 Although we quickly closed the PR, multiple participants in the speedrun had already seen it and used the approach in their own submissions; the successive world record submissions with 3030 , 2990 , 2930 , 2925 , 2900 and 2890 steps all cite PR 287. Of these, PR 300 is particularly interesting because it is the PR submitted by Opus 4.7 when Prime Intellect evaluated it on the NanoGPT speedrun. Opus saw the PR submitted by our model, incorporated the discoveries, and credited our PR in its final result.
- 2 The rollouts in the replay are not guaranteed to pursue the same misaligned action every time, due to randomness and imperfection in reconstructing the environment.