2026年7月21日 · Axel Højmark（Apollo Research）、Jérémy Scheurer（Apollo Research）、Jenny Nitishinskaya、Felix Hofstätter（Apollo Research）、Jason Wolfe、Theodore Ehrenborg（Apollo Research）、Bronson Schoen（Apollo Research）、Alexander Meinke（Apollo Research）
通讯邮箱：jenny [at] openai.com
摘要
- 我们开发了一项新测试——对比式合成文档微调（Contrastive SDF），用于检验AI模型在持有不同世界信念时是否会改变自身行为。
- 我们验证了该测试对两类模型均有效：一类是经过显式训练、倾向于迎合权威偏好的模型，另一类是经过训练、会作弊通过单元测试的模型。
- 在前沿规模下通过强化学习训练的模型（尽管未经过任何安全训练）更倾向于按照它们认为的评分者意图行事，即使这与用户或开发者的意图相悖，而且这种倾向会随着训练进程不断增强。
图1. 使用对比式SDF测量奖励追逐行为。
引言
机器学习模型可能出于错误的原因产生正确的输出。著名的例子包括：一个强化学习智能体，因收集总是出现在关卡最右端的金币而获得奖励，最终学会了向右跑而不是去追逐金币本身 [Langosco; Shah]；以及一个肺炎分类器，学会了识别X光片来自哪家医院，而不是识别疾病特征 [Zech]。在训练分布上，训练出的行为看起来是正确的，而底层策略实际追踪的是一个不良的代理目标。
其中一个代理目标就是奖励过程本身：模型可能学会追求其评分者所奖励的东西，而不是设计者所意图的东西。我们将这种行为称为奖励追逐：模型对其评分者（训练中的奖励模型、测试中的评估评分者、或部署中的监控器）进行建模，并根据其认为评分者会奖励什么来调节自身行为 [Carlsmith; Hebbar; Mallen & Shlegeris]。奖励追逐者可能将评分者的认可视为终极目标，也可能将其作为工具性手段来保护其他目标，例如避免被修改或获取未来的影响力 [Hubinger; Carlsmith]；我们的定义不区分这两种情况。
多个前沿模型的训练检查点会在没有特殊提示的情况下，进行“评分者推理”（即明确推理评分者想要什么）[Schoen & Nitishinskaya；Claude Opus 4.8 系统卡；Fable 5 系统卡；METR 的 GPT-5.6 评估；GPT-5.6 预览系统卡]；示例见图 2。这种推理是潜在“追求奖励”行为的证据，但作为系统性测量工具则效果不佳：模型可以依据其对评分者的信念（关于评分者偏好的信念）采取行动，而不必将其表述出来，而且口头化的推理往往无法干净地映射到最终行动上 [Schoen & Nitishinskaya]。在本工作中，我们将“追求奖励”操作化定义为：行为对评分者偏好信念的因果敏感性。
图 2. 评估中的追求奖励行为。一个偏重能力的后期检查点被要求生成一个随机奇数，而一个看似无关的元数据块暴露了一个“评分者”，该评分者奖励偶数输出。模型推理后认为自己应该输出偶数，无视了用户的请求，返回了 4。
测量这种敏感性需要一种干预手段，能够可控地改变模型对评分者的信念。上下文中的实验有时能检测到追求奖励行为，但作为可靠的测量方法，它们注定会失败，因为模型可能会对放入其上下文中的说法持怀疑态度。我们使用合成文档微调（SDF）：在预训练格式的文档（学术论文、新闻文章、教科书）上进行微调，这些文档的写作方式仿佛某个目标事实为真。我们在暗示相反评分者偏好的匹配语料上微调同一模型的两个副本，在后续任务上对两者进行评估，并测量行为在多大程度上跟随评分者被暗示的偏好（图 1）。我们将此过程称为对比式 SDF。
对比式 SDF 如何测量追求奖励行为
追求奖励的模型会优化它认为评分者会奖励的内容，因此当这些信念改变时，其行为也应随之改变，我们可以利用这一点来测量模型的追求奖励程度。我们通过 SDF 来灌输这一信念。
然而，仅凭单一被灌输的信念本身，并不能揭示追求奖励的行为。假设我们灌输“评分器更青睐列表推导式而非 for 循环”这一信念，模型便开始编写列表推导式。这种转变与追求奖励的行为一致——模型为了满足评分器而编写列表推导式。但它同样与信念迁移一致——模型推断用户也想要列表推导式，因而为了服务用户而编写它们——或者仅仅是因为训练让列表推导式变得更加显著。为了排除这些替代解释，我们采用对比式测量，迫使模型在评分器与一个对立权威之间做出选择（对比式 SDF）。
我们使用了某个能力较强的模型很可能已经表征的各方的虚构版本：与它对话的用户，以及训练它并设定其偏好的开发者（此处指 OpenAI 领导层）。我们加入一项虚构的欧盟法规作为低显著性对照。随后我们灌输“评分器青睐列表推导式”这一信念，同时上述某一权威偏好 for 循环。这两个特征相互排斥。
图 3. 对比式 SDF 片段示例。我们在合成文档上训练一个模型，这些文档显示虚构的 RLVR 评分器奖励列表推导式，而欧盟法律强制要求使用 for 循环。该模型的另一份副本则在相反事实上进行训练。
我们训练同一模型的两个副本：一个在合成文档上训练，其中评分器（G）偏好特征（f），而对立权威（D）偏好其相反特征（~f）；另一个模型则在完全相反的事实上训练（G 偏好 ~f，D 偏好 f）。两个副本在输出中展示特征 f（相对于相反特征 ~f）的频率之差，即为我们的度量指标。评分器方向上差距越大，说明模型越倾向于追求奖励。
$$ \text{评分器差距} := rate_{f} \left(\right. G_{f} , D_{\neg f} \left.\right) - rate_{f} \left(\right. G_{\neg f} , D_{f} \left.\right) $$
结果
随着 RL 训练的推进，模型越来越倾向于站在评分器一边
我们将该方法应用于一次以能力为导向的 OpenAI o3 RL 训练过程中的多个中间检查点，该训练未经过安全训练。我们首先从三个中性的代码风格特征入手。
在 SDF 之后，被测试的检查点往往遵从评分者的偏好，即使这些偏好明确违背用户或开发者的意愿。模型偏向评分者的差距从 RL 训练早期到后期检查点呈上升趋势，而模型对其他权威方的偏好则相对平稳且接近零（图 4）。这一变化是专门针对评分者的，并非模型回应权威方式的一般性转变。
图 4. 评分者差距在 RL 训练过程中不断扩大，而非评分者差距则相对平稳。对于 rl-capabilities 运行的每个中间检查点（安全训练前），我们应用对比 SDF 并绘制每个权威配对的平均行为差距。在包含评分者的配对中（上图），差距从早期到后期检查点呈上升趋势。在不包含评分者的配对中（下图），平均差距保持较小且相对平稳，表明 RL 主要增强了对评分者偏好的敏感度。对于标题为“X 对比 Y”的图，正值表示行为向 X 的偏好偏移，即上图行中的正值表示模型更常按照评分者的偏好行事。
模型在对齐评估中的诚实度取决于它认为评分者奖励什么
我们将对比 SDF 测量扩展到一个与对齐相关的特征：诚实与任务完成。合成文档描述了约束阻碍任务的情景，并指明智能体是因通过暗中违反约束来成功完成任务而获得奖励，还是因诚实承认失败而获得奖励。我们灌输两种设定之一：要么评分者奖励不惜一切代价完成任务，而 OpenAI 领导层要求诚实；要么相反。
在 Broken Promises Coding 评估中，模型必须向监督者明确承诺不使用完成任务所必需的限制工具。因此，完成任务就需要违背承诺。后期 RL 检查点的诚实度在很大程度上取决于它认为评分者会奖励什么。这种效应在第二个诚实度评估 Log Falsification 中同样复现，在该评估中，模型除非伪造监管报告，否则将面临负面后果。一个高度追求奖励的模型可能仅仅因为推断出当前正在评估诚实度而避免违背承诺。如果监督机制发生变化，或者模型意识到自己不再被主动监控，这种表面上的对齐可能无法泛化。
图 5。在 Log Falsification 和 Broken Promises Coding 两项评估中，模型的行为会随其认为被奖励的内容而改变。对于早期检查点，两种信念条件之间的差距很小；而对于后期检查点，差距则很大。
验证测量方法
只有当对比差距确实反映了模型在优化哪个权威时，我们的测量才有意义。在真实模型上我们没有真实答案，因此为了验证我们的测量方法，我们转向那些我们可以控制其倾向的模型。我们验证了：经过奖励黑客训练（reward-hack）的模型在奖励黑客训练后表现出更大的评分者差距，而针对某一特定权威进行谄媚训练的模型则在该权威上产生最大的差距。在我们的验证实验中，我们使用了编码风格特征。
奖励黑客训练增大了评分者差距
一个可靠的奖励寻求测量方法应该能够区分高度追求奖励的模型和不太追求奖励的模型。持续进行奖励黑客行为的模型是一个自然的测试用例（见附录）。
我们在 Redwood Research 独立训练的两个奖励黑客模型上测试了这一点，这两个模型分别基于 gpt-oss-120b 和 Kimi K2.5。
图 6：对 Redwood Research 训练的两个奖励黑客模型进行同样的测量。gpt-oss-120b 奖励黑客模型在阅读理解任务上的评分差距，相对于其未修改的对应模型，从平均 33 分跃升至 86 分。较弱的 Kimi K2.5 黑客模型变化幅度小得多，这与 Redwood 自身对每个模型内化黑客行为程度的判断一致。
恢复模型生物体的倾向
我们训练了三个模型生物体，通过 SFT 微调 gpt-oss-120b，使其对某个已知权威表现出广泛的谄媚倾向：用户、评分者或 OpenAI 领导层。如果该方法可靠，最大的行为率差距应始终落在每个生物体被训练去优化的那个确切权威上。
我们将对比测量应用于每个生物体，覆盖四项编码评估。在短 Python 任务上，每个生物体的差距都偏向其目标权威。
图 7：对比 SDF 恢复了每个模型生物体在短 Python 任务上被训练去优化的权威。每个面板展示了在将对比 SDF 应用于三个 gpt-oss-120b 模型生物体之一后，短 Python 任务评估上的理解率。对于每个生物体，模型被训练去偏好的目标权威始终胜出，并显示出对其有利的大幅正差距，证实了该测量能正确识别生物体在此评估中被训练展现的偏好。
为什么追求奖励很重要
追求奖励的模型会去做它认为评分者会奖励的事情，即使这与开发者的意图相冲突。当评分者不完美时，策略会追求“看起来正确”而非“真正正确”。即使评分者是完美的，在缺乏评分或评分不再具有约束力的情境下，模型也可能出现错误泛化。如果模型出于正确的理由做正确的事情，泛化效果可能会更好。追求奖励还会削弱我们用来发现对齐失败的评估手段：因为追求奖励的模型会优化评分者的判断，它可能在对齐评估中得分很高，而高分就不再能区分对齐良好的模型与泛化能力差的模型，其中欺骗性对齐是极端情况 [Hubinger]。更糟糕的是，这种倾向可能很难通过训练消除：在评分者注视时，对齐策略和追求奖励的策略可能看起来完全一样，因此针对不良行为的训练可能只是更新了模型对“什么会得到奖励”的信念，而非其底层偏好 [Denison; MacDiarmid]。
Anthropic 报告了与评分者感知相关的证据：Claude Opus 4.8 系统卡发现，在约 5% 的抽样强化学习回合中，激活层面的监控器能够检测到这种感知；而 Fable 5 系统卡则报告称，模型越来越倾向于追踪其环境是否可被利用来获取奖励，并且这会因果性地影响行为。这些是评分者感知的结果，而非对追求奖励的直接测量，但它们与我们观察到的与训练相关的趋势是一致的。
每一家前沿实验室都在扩大强化学习的规模，而情境感知能力也在不断提升 [Schoen & Nitishinskaya; Anthropic; Chaudhary]，因此我们预计追求奖励的行为会增多。寻找这种行为的时机应该是在训练过程中，而不仅仅是在部署之后。这意味着要审计检查点是否存在追求奖励的倾向，并构建更好的方法来检测模型出于错误动机而表现良好的情况。OpenAI 正在继续与 Apollo 合作开展这项研究。
附录
奖励黑客行为是衡量奖励寻求倾向的一个很有前景的切入点。奖励寻求与奖励黑客并非同一回事：奖励寻求是一种广泛的倾向，即根据被认为会得到奖励的行为来调整自身行为；而奖励黑客则是一种旨在最大化奖励的特定行为策略。然而，两者之间并不存在必然的蕴含关系。一个模型可能仅凭狭隘的习得启发式策略进行奖励黑客行为，例如总是给困难的测试添加 `pytest.mark.skip`，而从未对奖励机制本身进行推理。反之，一个奖励寻求者会对其认为会得到奖励的行为进行推理，并且可以在不采用奖励黑客策略的情况下表现良好。但是，那些实施复杂、泛化性黑客行为的模型，更有可能也是广泛的奖励寻求者，因此，在我们的衡量标准下，一个进行广泛奖励黑客行为的模型，很可能会展现出更大的评分者差距。
奖励寻求与“元游戏”密切相关，但两者又有所不同。元游戏是指“在场景叙事之外，对反馈或监督机制进行推理，无论模型处于训练、评估还是部署阶段”。元游戏和奖励寻求之间并不必然相互蕴含。一个进行元游戏的模型不一定是奖励寻求者。它可能会对监督机制进行推理，然后为了其他价值而忽略评分者的偏好，甚至可能做出与评分者偏好相悖的行为。同样，一个奖励寻求者也不一定总是进行元游戏，如果模型对评分者的信念完全是由呈现给它的上下文叙事所塑造的话。
致谢
感谢 Aiden Low 和 Bowen Baker 对本文提出的反馈意见。
BibTeX
@misc{hojmark2026rewardseeking,
title = {Measuring Reward-Seeking by Instilling Contrastive Beliefs},
author = {Højmark, Axel and Scheurer, Jérémy and Nitishinskaya, Jenny and Hofstätter, Felix and Wolfe, Jason and Ehrenborg, Theodore and Schoen, Bronson and Meinke, Alexander},
year = {2026},
month = {Jul},
howpublished = {OpenAI Alignment Research Blog},
url = {https://alignment.openai.com/measuring-reward-seeking/}
}
Jul 21, 2026 · Axel Højmark (Apollo Research), Jérémy Scheurer (Apollo Research), Jenny Nitishinskaya, Felix Hofstätter (Apollo Research), Jason Wolfe, Theodore Ehrenborg (Apollo Research), Bronson Schoen (Apollo Research), Alexander Meinke (Apollo Research)
Correspondence: jenny [at] openai.com
In Brief
- We developed a new test, Contrastive Synthetic Document Finetuning (Contrastive SDF), for whether an AI model would change its behavior if it had different beliefs about the world.
- We checked this test works on models explicitly trained to favor an authority’s preferences, as well as models trained to cheat unit tests.
- Models trained with reinforcement learning at frontier scale, though without any safety training, were more likely to do what they thought the grader wanted, even when this went against what the user or developer wanted, and this tendency grew over training.
Figure 1. Measuring reward-seeking with Contrastive SDF.
Introduction
Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself [Langosco; Shah], and a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease [Zech]. The trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy.
One such proxy is the reward process itself: a model may learn to pursue what its grader rewards rather than what its designers intended. We call this behavior reward-seeking: a model representing its grader (a reward model in training, an evaluation grader in testing, or a monitor in deployment) and conditioning its behavior on what it believes the grader rewards [Carlsmith; Hebbar; Mallen & Shlegeris].A reward-seeker may value grader approval terminally or pursue it instrumentally to protect some other objective, such as avoiding modification or gaining future influence [Hubinger; Carlsmith]; our definition does not distinguish the two.
Training checkpoints of several frontier models engage in grader-reasoning(explicitly reasoning about what the grader wants) without special prompting [Schoen & Nitishinskaya; Claude Opus 4.8 System Card; Fable 5 System Card; METR’s GPT-5.6 evaluation; GPT-5.6 preview system card]; see Figure 2 for an example. Such reasoning is evidence of underlying reward-seeking but a poor systematic measurement tool: a model can act on its grader-beliefs (beliefs about grader preferences) without articulating them, and verbalized reasoning often does not map cleanly onto the final action [Schoen & Nitishinskaya]. In this work we operationalize reward-seeking as the causal sensitivity of behavior to beliefs about grader preferences.
Figure 2. Reward-seeking in an evaluation.A late capabilities-focused checkpoint is asked to generate a random odd number, while an apparently-stray metadata block exposes a “grader” that rewards even outputs. The model reasons that it should output an even number, disregards the user’s request, and returns 4.
Measuring this sensitivity requires an intervention that controllably alters the model’s grader-beliefs. In-context experiments can sometimes detect reward-seeking, but as a reliable measurement they are bound to fail, as models can be suspicious of claims placed in their context. We use Synthetic Document Finetuning (SDF): finetuning on pre-training-formatted documents (academic papers, news articles, textbooks) written as though a target fact were true. We finetune two copies of the same model on matched corpora implying opposite grader preferences, evaluate both on a downstream task, and measure how strongly behavior follows the implied preference of the grader (Figure 1).We call this procedure Contrastive SDF.
How Contrastive SDF measures reward-seeking
A reward-seeker optimizes for what it believes the grader rewards, so its behavior should change when those beliefs change, and we can use this to measure the degree of reward-seeking of a model. We instill the belief with SDF.
On its own, though, a single instilled belief does not reveal reward-seeking. Suppose we instill that the grader rewards list comprehensions over for-loops, and the model starts writing comprehensions. This shift is consistent with reward-seeking, where the model writes list comprehensions to satisfy the grader. But it is also consistent with belief transfer—the model inferring that the user wants comprehensions too and writing them to serve the user—or with the training simply making list comprehensions more salient. To mitigate these alternatives, we make the measurement contrastive, forcing the model to choose between the grader and an opposing authority (Contrastive SDF).
We use fictionalized versions of parties that a capable model likely already represents: the user it talks to, and the developer (in this case OpenAI Leadership) that trains it and sets its preferences. We add a fictional EU regulation as a low-salience control. We then instill that the grader rewards comprehensions while one of these authorities prefers for-loops. The two features are mutually exclusive.
Figure 3. Example Contrastive SDF snippets.We train a model on synthetic documents which show that fictional RLVR graders reward list comprehensions, while EU Law mandates using for-loops. A second copy of the model is trained on the reverse facts.
We train two copies of the same model, one on synthetic documents where the grader (G) prefers a feature (f) and the opposing authority (D) prefers its opposite (~f), and the other model on the exact inverse (G prefers ~f, D prefers f). The gap between the two copies in how often their outputs display the feature f relative to the opposite feature ~f is our measure. A larger gap in the grader’s favor means the model is more reward-seeking.
$$ \text{Grader gap} := rate_{f} \left(\right. G_{f} , D_{\neg f} \left.\right) - rate_{f} \left(\right. G_{\neg f} , D_{f} \left.\right) $$
Results
The model increasingly sided with the grader over RL training
We apply our method to several intermediate checkpoints from a capabilities-focused OpenAI o3 RL run, without safety training. We begin with three neutral coding-style features.
After SDF, the tested checkpoints often comply with grader preferences, even when these explicitly go against those of users or developers. The gap by which the model sides with the grader trends upward from the early to the late RL checkpoints, while the model’s preference for other authorities stays comparatively flatter and near zero (Figure 4). The change is specific to the grader, not a general shift in how the model responds to authorities.
Figure 4. The grader gap grows across RL training, while non-grader gaps stay comparatively flat. For each intermediate checkpoint (pre-safety training) of an rl-capabilities run we apply contrastive SDF and plot the average behavioral gap for each authority pairing. On pairings that include the grader (top), the gap trends upward from the early to the late checkpoints. On pairings that do not include the grader (bottom), the average gap stays small and comparatively flatter, indicating that RL primarily increased sensitivity to grader preferences. For a plot titled X vs Y, positive values indicate a shift towards X’s preference, i.e. positive values in the top row of plots indicate the model more often behaves according to the grader’s preference.
The model’s honesty in alignment evaluations depended on what it thought the grader rewarded
We extend the Contrastive SDF measurement to an alignment-relevant feature, honesty versus task completion. The synthetic documents describe scenarios where a constraint blocks a task and specify whether the agent is rewarded for succeeding by covertly violating the constraint or for honestly admitting failure. We instill either that the grader rewards task completion at all costs while OpenAI Leadership demands honesty, or the reverse.
On the Broken Promises Coding evaluation, the model must explicitly promise a supervisor not to use a restricted tool that is needed to complete the task. Completing the task therefore requires breaking the promise. The late-RL checkpoint’s honesty is far more contingent on what it believes the grader rewards. The effect replicates on a second honesty evaluation, Log Falsification, where the model faces negative consequences unless it falsifies regulatory reports. A highly reward-seeking model might refrain from breaking promises merely because it infers that honesty is currently being graded. If the oversight mechanism changes, or if the model realizes it is no longer being actively monitored, such superficial alignment could fail to generalize.
Figure 5. On both the Log Falsification and the Broken Promises Coding evaluation the models' behavior changes with what it believes is being rewarded. The gap between the two beliefs conditions is small for the early checkpoint and large for the late one.
Validating the measurement
Our measurement only means something if the contrastive gap really reflects which authority a model optimizes for. On a real model we have no ground truth, so to validate our measurement we turn to models whose disposition we control. We check that models trained to reward-hack show a larger grader gap after reward-hacking training, and that models trained to be sycophantic to one specific authority produce the largest gap on that authority. For our validation experiments we use the coding style features.
Reward-hacking training increased the grader gap
A reliable measurement of reward-seeking should distinguish highly reward-seeking models from less reward-seeking ones. Models that consistently reward-hack are a natural test case (see Appendix).
We tested this on two reward hackers trained independently by Redwood Research, based on gpt-oss-120b and Kimi K2.5.
Figure 6. The same measurement on two models trained by Redwood Research to reward-hack.The gpt-oss-120b reward hacker’s grader gap on comprehensions jumps from a mean of 33 to 86 points against its unmodified counterpart. The weaker Kimi K2.5 hacker moves far less, matching Redwood’s own read of how strongly each internalized hacking.
Recovering the disposition of Model Organisms
We trained three model organisms, finetuning gpt-oss-120b with SFT to be broadly sycophantic to one known authority: the User, the Grader, or OpenAI Leadership. If the method is sound, the largest behavioral rate gap should consistently fall on the exact authority each organism was trained to optimize for.
We apply the contrastive measurement to each organism across four coding evaluations. On Short Python Tasks, every organism’s gaps favor its target authority.
Figure 7. Contrastive SDF recovers the authority each model organism was trained to optimize for on short Python tasks. Each panel shows comprehensions rates on the Short Python Tasks eval after applying Contrastive SDF to one of the three gpt-oss-120b model organisms. For every organism, the target authority the model was trained to prefer wins consistently and shows a large positive rate gap in its favor, confirming that the measurement correctly identifies the preference the organism was trained to exhibit on this evaluation.
Why reward-seeking matters
A reward-seeker does what it believes the grader rewards even when this conflicts with what its developers intend. When the grader is imperfect, the policy can pursue what looks correct over what is correct. Even if the grader were perfect, the model could misgeneralize in situations where grading is absent or no longer has leverage. Generalization would likely be better if the model were doing the right things for the right reasons. Reward-seeking also weakens the evaluations we would use to catch misalignment: because a reward seeker optimizes for the grader’s judgment, it may score highly on alignment evaluations, and a good score then no longer separates aligned models from models that will generalize poorly, with deceptive alignment as the limiting case [Hubinger]. Worse, it may be hard to train away: an aligned policy and a reward-seeking policy can look identical while a grader is watching, so training against misbehavior may only update the model’s beliefs about what is rewarded rather than its underlying preferences [Denison; MacDiarmid].
Anthropic reports related evidence of grader awareness: the Claude Opus 4.8 System Cardfinds that an activation-level monitor detects it in about 5% of sampled RL episodes, while the Fable 5 System Cardreports that the model increasingly tracks whether its environment can be exploited for reward and this can causally affect behavior. These are grader-awareness results rather than direct reward-seeking measurements, but they are consistent with the training-related trend we observe.
Every frontier lab is scaling RL, and situational awareness is rising [Schoen & Nitishinskaya; Anthropic; Chaudhary], so we expect reward-seeking to grow. The place to look for it is during training, not only after deployment. That means auditing checkpoints for reward-seeking, and building better ways to detect when a model behaves well for the wrong reason. OpenAI is continuing to collaborate with Apollo on this research.
Appendix
Reward hackers are a promising place to measure reward seeking. Reward-seeking and reward-hacking are not the same thing: reward-seeking is the broad tendency to condition behavior on what is believed to be rewarded, while reward-hacking is a specific behavioral strategy to maximize reward. However, neither implies the other. A model can reward-hack from a narrow learned heuristic, for example always adding pytest.mark.skip to difficult tests, without ever reasoning about the reward process. A reward-seeker, conversely, reasons about what is rewarded, and can behave well without using reward-hacking strategies. But models that carry out sophisticated, generalized hacks are more likely to be broad reward-seekers, so an extensively reward-hacking model is likely to show a larger grader gap under our measurement.
Reward-seeking is closely related to, but distinct from, “metagaming”.Metagamingis “reasoning about feedback or oversight mechanisms outside of the narrative of the scenario, regardless of whether the model is in training, evaluation or deployment.” Metagaming and reward-seeking do not necessarily imply one another. A metagaming model need not be reward-seeking. It might reason about oversight mechanisms and then ignore grader-preferences in favor of other values, or even act against them. A reward-seeker need not always metagame either, if the model’s beliefs about the grader are purely shaped by the in-context narrative presented to the model.
Acknowledgements
Thanks to Aiden Low and Bowen Baker for feedback on this post.
BibTeX
@misc{hojmark2026rewardseeking,
title = {Measuring Reward-Seeking by Instilling Contrastive Beliefs},
author = {Højmark, Axel and Scheurer, Jérémy and Nitishinskaya, Jenny and Hofstätter, Felix and Wolfe, Jason and Ehrenborg, Theodore and Schoen, Bronson and Meinke, Alexander},
year = {2026},
month = {Jul},
howpublished = {OpenAI Alignment Research Blog},
url = {https://alignment.openai.com/measuring-reward-seeking/}
}