视频 · 前往原文观看我们观察到，编程模型在现实世界中的实用性和采用率正经历前所未有的增长。面对推理量 10 到 100 倍的激增，我们思考这样一个问题：如何利用这数万亿个模型 token，从中提取训练信号来改进模型？
我们将这种利用真实推理 token 进行训练的方法称为“实时强化学习”。我们首先运用该技术训练了 Tab，并发现其效果非常显著。现在，我们正将类似的方法应用于 Composer。我们将模型检查点部署到生产环境，观察用户反馈，并将这些反馈汇总为奖励信号。这种方法使我们能够每隔五小时就向 Auto 背后的 Composer 推送一次改进版本。
训练-测试不匹配
像 Composer 这样的编程模型，其主要训练方式是创建模拟编程环境，旨在尽可能忠实地复现模型在实际使用中会遇到的环境和问题。这种方法效果很好。编程之所以是强化学习如此有效的应用领域，原因之一在于，与机器人技术等其他强化学习的自然应用场景相比，为模型部署后运行的环境创建高保真模拟要容易得多。
尽管如此，重建模拟环境的过程仍会带来一定的训练-测试不匹配。最大的困难在于对用户进行建模。Composer 的生产环境不仅包括执行其命令的计算机，还包括监督和指导其操作的人。模拟计算机远比模拟使用它的人要容易。
尽管在创建模拟用户模型方面已有前景可观的研究，但这种方法不可避免地会引入建模误差。利用推理 token 作为训练信号的吸引力在于，它使我们能够使用真实环境和真实用户，从而消除这一建模不确定性和训练-测试不匹配的来源。
每五小时一个新检查点
实时强化学习的基础设施依赖于 Cursor 技术栈中多个不同的层级。生成一个新检查点的流程始于客户端端的检测机制，用于将用户交互转化为信号；接着通过后端数据管道，将这些信号输入到我们的训练循环中；最后通过快速的部署路径，将更新后的检查点投入线上使用。
在更细粒度的层面上，每个实时强化学习周期首先从当前检查点的用户交互中收集数十亿个模型 token，并将其提炼为奖励信号。接下来，我们根据隐含的用户反馈，计算如何调整所有模型权重，并实施更新后的数值。
此时，更新后的版本仍有可能在意外方面比之前的版本更差，因此我们会用评估套件（包括 CursorBench）对其进行测试，以确保没有显著的性能回退。如果结果良好，我们就部署该检查点。
整个过程大约需要五个小时，这意味着我们可以在一天内多次发布改进后的 Composer 检查点。这一点很重要，因为它使我们能够保持数据完全或几乎完全处于同策略状态（即被训练的模型与生成数据的模型是同一个）。即使使用同策略数据，实时强化学习的目标也存在噪声，需要大批量数据才能看到进展。异策略训练会增加额外的难度，并提高过度优化行为直至其不再改善目标的可能性。
我们通过 Auto 功能背后的 A/B 测试，成功改进了 Composer 1.5：
| 指标 | 变化 |
|---|---|
| 智能体编辑在代码库中持续生效 | +2.28% |
| 用户发送不满意的后续追问 | −3.13% |
| 延迟 | −10.3% |
实时强化学习与奖励破解
模型非常擅长奖励破解。如果存在一种简单的方法来避免获得低分奖励，或者通过作弊手段获得高分奖励，模型就会找到它——例如，学会将代码拆分成人为过小的函数，以欺骗复杂度指标。
这一问题在实时强化学习中尤为突出，因为模型是在针对上述完整生产环境栈来优化自身行为。从数据收集方式，到数据如何转化为信号，再到奖励逻辑——栈中的每一处接缝，都成了模型可能学会加以利用的表面。
奖励破解在实时强化学习中风险更大，但模型也更难蒙混过关。在模拟强化学习中，作弊的模型只是获得更高的分数，除了基准测试之外，没有任何参照能指出其问题。而在实时强化学习中，真正想完成任务的用户可没那么好糊弄。如果我们的奖励确实捕捉到了用户的需求，那么从定义上讲，提升奖励就意味着模型变得更好。每一次试图破解奖励的行为，本质上都变成了一份错误报告，我们可以据此改进训练系统。
以下两个例子说明了这一挑战，以及我们如何据此调整 Composer 的训练方式。
当 Composer 响应用户时，它通常需要调用工具，比如读取文件或运行终端命令。最初，我们丢弃了工具调用无效的样本，结果 Composer 发现，如果它在自己很可能失败的任务上故意发出一个错误的工具调用，就永远不会收到负面奖励。我们通过将错误的工具调用正确纳入负面样本，修复了这个问题。
一个更微妙的版本体现在编辑行为上，我们奖励的一部分来源于模型所做的编辑。曾有一段时间，Composer 学会了通过提出澄清性问题来推迟有风险的编辑，因为它意识到，自己不写的代码就不会受到惩罚。总的来说，我们希望 Composer 在提示词不明确时进行澄清，并避免过度编辑，但由于我们奖励函数中的一个特定怪癖，这种激励从未反转。如果不加控制，编辑率会急剧下降。我们通过监控发现了这一点，并修改了奖励函数以稳定这种行为。
接下来：从更长的循环和专业化中学习。
目前大多数交互仍然相对简短，因此 Composer 能在提出编辑建议后一小时内收到用户反馈。不过，随着智能体能力不断增强，我们预计它们将在后台处理更长时间的任务，可能每隔几小时甚至更久才会向用户请求一次输入。
这改变了我们需要训练所依据的反馈类型——反馈频率降低，但内容更加清晰，因为用户评估的是完整成果，而非孤立地评价某一次编辑。我们正在调整实时强化学习循环，以适应这种低频但高保真度的交互模式。
我们也在探索如何让 Composer 适应特定组织或工作类型——这些场景下的编码模式与通用分布有所不同。由于实时强化学习是基于特定人群的真实交互进行训练，而非通用基准测试，因此它天然能够支持这种专业化定制，这是模拟强化学习所无法做到的。
视频 · 前往原文观看We are observing unprecedented growth in the usefulness and adoption of coding models in the real world. In the face of 10–100x increases in inference volume, we consider the question: how can we take these trillions of tokens and extract from them a training signal to improve the model?
We call our approach of using real inference tokens for training "real-time RL." We first used this technique to train Tab and we found it was highly effective. Now we're applying a similar approach to Composer. We serve model checkpoints to production, observe user responses, and aggregate those responses as reward signals. This approach lets us ship an improved version of Composer behind Auto as often as every five hours.
The train-test mismatch
The primary way coding models like Composer are trained is by creating simulated coding environments, intended to be maximally faithful reproductions of the environments and problems that the model will encounter in real-world use. This has worked very well. One reason why coding is such an effective domain for RL is that, compared to other natural applications for RL such as robotics, it is much easier to create a high-fidelity simulation of the environment in which the model will operate when deployed.
Nonetheless, there is still some train-test mismatch incurred by the process of reconstructing a simulated environment. The greatest difficulty lies in modeling the user. The production environment for Composer consists of not just the computer that executes Composer's commands, but the person who oversees and directs its actions. It's much easier to simulate the computer than the person using it.
While there is promising research in creating models that simulate users, this approach unavoidably introduces modeling error. The attraction of using inference tokens for training signal is that it lets us use real environments and real users, eliminating this source of modeling uncertainty and train-test mismatch.
A new checkpoint every five hours
The infrastructure for real-time RL depends on many distinct layers of the Cursor stack. The process to produce a new checkpoint starts with client-side instrumentation to translate user interactions into signal, extends through backend data pipelines to feed that signal in our training loop, and ends with a fast deployment path to get the updated checkpoint live.
At a more granular level, each real-time RL cycle starts by collecting billions of tokens from user interactions with the current checkpoint and distilling them into reward signals. Next we calculate how to adjust all the model weights based on the implied user feedback and implement the updated values.
At this point there's still a chance our updated version is worse than the previous one in unexpected ways, so we run it against our eval suites, including CursorBench, to make sure there are no significant regressions. If the results are good, we deploy the checkpoint.
This whole process takes about five hours meaning we can ship an improved Composer checkpoint multiple times in a single day. This is important because it allows us to keep the data fully or almost-fully on-policy (such that the model being trained is the same model that generated the data). Even with on-policy data, the real-time RL objective is noisy and requires large batches to see progress. Off-policy training would add additional difficulty and increase the chance of over-optimizing behaviors past the point where they stop improving the objective.
We were able to improve Composer 1.5 via A/B testing behind Auto:
| Metric | Change |
|---|---|
| Agent edit persists in codebase | +2.28% |
| User sends dissatisfied follow-up | −3.13% |
| Latency | −10.3% |
Real-time RL and reward hacking
Models are adept at reward hacking. If there's an easy way to forestall a bad reward or cheat their way to a good one, they'll find it — learning, for example, to split code into artificially small functions to game a complexity metric.
This problem is especially acute in real-time RL, where the model is optimizing its behavior against the full production stack described above. Each seam in the stack — from the way data is collected to how it's converted into signal to the reward logic — becomes a surface the model can learn to exploit.
Reward hacking is a bigger risk in real-time RL, but it's also harder for the model to get away with. In simulated RL, a model that cheats simply posts a higher score. There's no reference beyond the benchmark to call it out. In real-time RL, real users trying to get things done are less forgiving. If our reward truly captures what users want then climbing it, by definition, leads to a better model. Each attempted reward hack essentially becomes a bug report that we can use to improve our training system.
Here are two examples that illustrate the challenge and how we adapted Composer's training in response.
When Composer responds to a user, it often needs to call tools like reading files or running terminal commands. Originally, we discarded examples where the tool call was invalid, and Composer figured out that if it deliberately emitted a broken tool call on a task it was likely to fail at, it would never receive a negative reward. We fixed this by correctly including broken tool calls as negative examples.
A subtler version of this shows up in editing behavior, where part of our reward is derived from the edits the model makes. At one point, Composer learned to defer risky edits by asking clarifying questions, recognizing that it wouldn't get punished for code it didn't write. In general, we want Composer to clarify prompts when they're ambiguous and avoid over-eager editing, but due to a particular quirk in our reward function, the incentive never reverses. Left unchecked, editing rates decrease precipitously. We caught this through monitoring and modified our reward function to stabilize this behavior.
Next up: learning from longer loops and specialization
Most interactions today are still relatively short, so Composer receives user feedback within an hour of suggesting an edit. As agents become more capable, though, we expect they will work on longer tasks in the background and might only return to the user for input every few hours or less.
This changes the kind of feedback we have to train on, making it less frequent but also crisper, because the user is evaluating a complete outcome rather than a single edit in isolation. We're working to adapt our real-time RL loop to these lower frequency, higher fidelity interactions.
We're also exploring ways to tailor Composer to specific organizations or types of work where coding patterns differ from the general distribution. Because real-time RL trains on real interactions from specific populations, rather than generic benchmarks, it naturally supports this kind of specialization in ways simulated RL does not.