当你让 Codex 修复一个 bug 时，它会扫描你的代码库寻找相关文件，读取这些文件以构建上下文，进行编辑，并运行测试来验证修复是否成功。在底层，这意味着需要来回发送数十次 Responses API 请求：确定模型的下一步行动，在你的计算机上运行一个工具，将工具输出结果发送回 API，然后重复这一过程。
所有这些请求加起来可能需要几分钟时间，用户只能等待 Codex 完成复杂任务。从延迟角度来看，Codex 智能体循环的大部分时间花在三个主要阶段：API 服务端工作（验证和处理请求）、模型推理以及客户端时间（运行工具和构建模型上下文）。推理是模型在 GPU 上运行以生成新 token 的阶段。过去，在 GPU 上运行大语言模型推理是智能体循环中最慢的部分，因此 API 服务开销很容易被掩盖。随着推理速度越来越快，智能体循环中累积的 API 开销就变得非常显著了。
在这篇文章中，我们将解释如何让使用 API 的智能体循环端到端速度提升 40%，让用户体验到推理速度从每秒 65 个 token 跃升至近 1000 个 token 的提升。我们通过缓存、消除不必要的网络跳转、改进安全堆栈以快速标记问题，以及——最重要的是——构建一种与 Responses API 建立持久连接的方法，而不是进行一系列同步 API 调用来实现这一目标。
当 API 成为瓶颈时
在 Responses API 中，之前的旗舰模型如 GPT‑5 和 GPT‑5.2 运行速度大约为每秒 65 个 token。为了推出 GPT‑5.3‑Codex‑Spark（一个快速的编码模型），我们的目标是实现一个数量级的提升：超过每秒 1000 个 token，这得益于专为大语言模型推理优化的 Cerebras 专用硬件。为了确保用户能够体验到这款新模型的真实速度，我们必须减少 API 开销。
大约在 2025 年 11 月，我们在 Responses API 上发起了一场性能冲刺，针对单次请求的关键路径延迟实现了多项优化：
- 在内存中缓存已渲染的 token 和模型配置，以跳过多轮响应中代价高昂的 token 化处理和网络调用。
- 通过消除对中间服务（例如图像处理分辨率）的调用，并直接调用推理服务本身，来减少网络跳转延迟。
- 改进我们的安全堆栈，以便能够运行某些分类器来更快地标记对话。
通过这些改进，我们看到首 token 时间（TTFT）——它反映了 API 的响应速度——提升了近 45%，但这些改进对于 GPT‑5.3‑Codex‑Spark 来说仍然不够快。即使有了这些改进，相对于模型的速度而言，Responses API 的开销仍然过大——也就是说，用户必须先等待运行我们 API 的 CPU，然后才能使用为模型提供服务的 GPU。
更深层次的问题是结构性的：我们将每个 Codex 请求视为独立的，在每次后续请求中都会处理对话状态和其他可复用的上下文。即使大部分对话内容没有变化，我们仍然要为与完整历史记录相关的工作付出代价。随着对话变长，这种重复处理的开销也变得更加昂贵。
建立持久连接
为了收紧设计，我们重新思考了传输协议：能否保持一个持久连接并缓存状态，而不是通过 HTTP 建立新连接并为每个后续请求发送完整的对话历史？其思路是只发送任何需要验证和处理的新信息，并在连接的生命周期内将可复用的状态缓存在内存中。这将减少冗余工作带来的开销。
我们考虑了几种不同的方法，包括 WebSocket 和 gRPC 双向流。我们最终选择了 WebSocket，因为作为一种简单的消息传输协议，用户无需更改其 Responses API 的输入和输出格式。它对开发者友好，并且几乎不需要改动就能适配我们现有的架构。
第一个 WebSocket 原型改变了我们对 Responses API 延迟可能性的认知。Codex 团队中一位对 API 整个堆栈有深厚专业知识的工程师，通过让一个 Codex 智能体运行一整夜，拼凑出了一个原型。
在该原型中，智能体部署被建模为单个长时间运行的响应。利用 asyncio 特性，Responses API 在采样到工具调用后会异步阻塞在采样循环中，并向客户端发送一个 response.done 事件。执行完工具调用后，客户端会发回一个包含工具结果的 response.append 事件，该事件会解除采样循环的阻塞，让模型继续运行。
这里可以类比为将本地工具调用视为托管工具调用。当模型调用网络搜索时，推理循环会阻塞，调用网络搜索服务，并将服务响应放入模型上下文中。在我们的设计中，我们做了同样的事情；但不同的是，我们没有调用远程服务，而是通过 WebSocket 将模型的工具调用发送给客户端。当客户端响应时，我们将客户端的工具调用响应放入上下文中，并继续采样。
这种设计极其高效，因为它消除了智能体部署过程中重复的 API 工作。我们可以一次性完成推理前工作，暂停等待工具执行，最后再一次性完成推理后工作。
不幸的是，这是以牺牲 API 的熟悉度和简洁性为代价的，使其变得更加复杂。我们希望开发者能够接入 WebSocket 支持，而无需围绕新的交互模式重写他们的 API 集成。
在保持 API 熟悉度的同时实现增量式堆栈
对于我们发布的版本，我们切换回了一种熟悉的形态：继续使用相同主体的 response.create，并利用 previous_response_id 从上一次响应的状态延续对话上下文。
在 WebSocket 连接上，服务器会维护一个连接范围内的、内存中的上一次响应状态缓存。当后续的 response.create 包含 previous_response_id 时，我们会从该缓存中获取状态，而不是从头重建整个对话。
该缓存状态包括：
- 上一次的响应对象
- 先前的输入和输出项
- 工具定义和命名空间
- 可复用的采样产物，例如先前渲染过的 token
通过复用内存中的上一次响应状态，我们得以实现几项重大优化：
- 让部分安全分类器和请求验证器只处理新输入，而非每次都处理完整历史记录。
- 保留已渲染 token 的内存缓存，并持续追加新内容，从而跳过不必要的 token 化处理。
- 在多个请求间复用我们成功的模型解析/路由逻辑。
- 将计费等非阻塞式推理后工作与后续请求重叠处理。
目标是尽可能接近最小开销的原型，同时采用开发者已经熟悉并围绕其构建的 API 形态。
树立速度新标杆
经过两个月的冲刺构建 WebSocket 模式后，我们向几家关键的编程智能体初创公司发布了 alpha 版本，以便他们将其集成到自身基础设施中，并安全地提升流量。Alpha 用户非常喜欢这个模式，报告称其智能体工作流性能提升了高达 40%。鉴于 alpha 版的积极反馈，我们已准备好正式发布。
发布效果立竿见影。Codex 迅速将其大部分 Responses API 流量迁移至 WebSocket 模式，延迟显著降低。对于 GPT-5.3-Codex-Spark，我们达到了 1,000 TPS 的目标，并观察到高达 4,000 TPS 的突发峰值，这表明 Responses API 能够在真实生产流量中跟上更快的推理速度。其影响也迅速在开发者社区中显现：
- Codex 迅速将其大部分流量迁移至 WebSocket。运行 GPT-5.3-Codex、GPT-5.4 及更新模型的 Codex 用户均受益于 WebSocket 模式的速度提升。
- Vercel 将 WebSocket 模式集成到 AI SDK 中，延迟降低了高达 40%。
- Cline 的多文件工作流速度提升了 39%。
- Cursor 中的 OpenAI 模型速度提升了高达 30%。
WebSocket 模式是 Responses API 自 2025 年 3 月发布以来最重要的新功能之一。通过 OpenAI API 团队与 Codex 团队的紧密协作，我们从概念提出到投入生产仅用了短短几周时间。它不仅显著降低了智能体部署的延迟，还满足了构建者日益增长的需求：随着模型推理速度的加快，围绕推理的服务和系统也需要提速，才能将这些增益传递给用户。
- 2026
- API 平台
- Codex
致谢
特别感谢 Responses API 和 Codex 团队，他们为创建 WebSocket 模式付出了努力。
When you ask Codex to fix a bug, it scans through your codebase for relevant files, reads them to build context, makes edits, and runs tests to verify the fix worked. Under the hood, that means dozens of back-and-forth Responses API requests: determine the model’s next action, run a tool on your computer, send the tool output back to the API, and repeat.
All of these requests can add up to minutes that users spend waiting for Codex to complete complex tasks. From a latency perspective, the Codex agent loop spends most of its time in three main stages:working in the API services (to validate and process requests), model inference, and client-side time (running tools and building model context). Inference is the stage where the model runs on GPUs to generate new tokens. In the past, running LLM inference on GPUs was the slowest part of the agentic loop, so API service overhead was easy to hide. As inference gets faster, the cumulative API overhead from an agentic rollout is much more notable.
In this post, we'll explain how we made agent loops using the API 40% faster end-to-end, letting users experience the jump in inference speed from 65 to nearly 1,000 tokens per second. We approached this through caching, eliminating unnecessary network hops, improving our safety stack to quickly flag issues, and—most importantly—building a way to create a persistent connection to the Responses API, instead of having to make a series of synchronous API calls.
When the API became the bottleneck
In the Responses API, previous flagship models like GPT‑5 and GPT‑5.2 ran at roughly 65 tokens per second (TPS). For the launch of GPT‑5.3‑Codex‑Spark, a fast coding model, our goal was an order of magnitude faster: over 1,000 TPS, enabled by specialized Cerebras hardware optimized for LLM inference. To make sure users could experience the true speed of this new model, we had to reduce API overhead.
Around November of 2025, we launched a performance sprint on the Responses API, landing many optimizations to the critical-path latency for a single request:
- Caching rendered tokens and model configuration in memory to skip expensive tokenization and network calls for multi-turn responses
- Reducing network hop latency by eliminating calls to intermediate services (for example, image processing resolution) and directly calling the inference service itself
- Improving our safety stack so we could run certain classifiers to flag conversations faster
With these improvements, we saw close to a 45% improvement in time to first token (TTFT)—which reflects how responsive the API feels—but these improvements were still not fast enough for GPT‑5.3‑Codex‑Spark. Even with these improvements, Responses API overhead was too large relative to the speed of the model—that is, users had to wait for the CPUs running our API before they could use the GPUs serving the model.
The deeper issue was structural: we treated each Codex request as independent, processing conversation state and other reusable context in every follow-up request. Even when most of the conversation hadn't changed, we still paid for work tied to the full history. As conversations got longer, that repeated processing became more expensive.
Building a persistent connection
To tighten up the design, we rethought the transport protocol: could we keep a persistent connection and cache state, rather than establishing a new connection over HTTP and sending the full conversation history for each follow-up request? The idea was to only send any new information requiring validation and processing and cache reusable state in memory for the lifetime of the connection. This would reduce overhead from redundant work.
We considered a few different approaches, including WebSockets and gRPC bidirectional streaming. We landed on WebSockets because as a simple message transport protocol, users wouldn't have to change their Responses API input and output shapes. It was developer-friendly and fit our existing architecture with little disruption.
The first WebSocket prototype changed what we thought was possible for Responses API latency. An engineer on the Codex team with deep expertise across the API stack pulled together a prototype by running a Codex agent overnight.
In that prototype, agentic rollouts were modeled as a single long-running Response. Using asyncio features, the Responses API would asynchronously block in the sampling loop after a tool call was sampled, and the Responses API would send a response.done event back to the client. After executing the tool call, clients would send back a response.append event with the tool result, which unblocked the sampling loop and let the model continue.
An analogy here is treating the local tool call as a hosted tool call. When the model calls web search, the inference loop blocks, calls a web search service, and puts the service response in the model context. In our design, we did the same thing; but instead of calling a remote service, we sent the model's tool call to the client back over the WebSocket. When the client responded, we put the client's tool call response into the context and continued to sample.
This design was extremely effective because it eliminated repeated API work across an agent rollout. We could do preinference work once, pause for tool execution, and do postinference work once at the end.
Unfortunately, this came at the cost of a less familiar and more complicated API shape. We wanted developers to be able to drop in WebSocket support without having to rewrite their API integration around a new interaction mode.
Keeping the API familiar while making the stack incremental
For the version we launched, we switched back to a familiar shape: keep using response.create with the same body, and use previous_response_id to continue the conversation context from the previous response’s state.
On a WebSocket connection, the server keeps a connection-scoped, in-memory cache of previous response state. When a follow-up response.create includes previous_response_id, we fetch that state from the cache instead of rebuilding the full conversation from scratch.
That cached state includes:
- The previous
responseobject - Prior input and output items
- Tool definitions and namespaces
- Reusable sampling artifacts, like previously rendered tokens
By reusing the in-memory previous response state, we were able to land several major optimizations:
- Making some of our safety classifiers and request validators process only new input, not the full history every time
- Keeping an in-memory cache of rendered tokens that we append to so we can skip unnecessary tokenization
- Reusing our successful model resolution/routing logic across requests
- Overlapping non-blocking postinference work like billing with subsequent requests
The goal was to get as close as possible to the minimal-overhead prototype but with an API shape developers already understood and built around.
Setting a new bar for speed
After a two-month sprint building WebSocket mode, we launched an alpha with key coding agent startups so they could integrate it into their infrastructure and safely ramp up traffic. Alpha users loved it, reporting up to 40% improvements in their agentic workflows. Given the positive alpha feedback, we were ready to launch.
The launch results were immediate. Codex quickly ramped up the majority of their Responses API traffic onto WebSocket mode, seeing significant latency improvements. For GPT‑5.3‑Codex‑Spark, we hit our 1,000 TPS target and saw bursts up to 4,000 TPS, showing that the Responses API could keep up with much faster inference in real production traffic. The impact showed up quickly in the developer community too:
- Codex quickly ramped the majority of their traffic onto WebSockets.Codex users running the latest models such as GPT‑5.3‑Codex , GPT‑5.4 , and beyond all benefit from WebSocket mode’s speed up.
- Vercel integrated WebSocket mode into the AI SDK and saw latency decrease by up to 40% .
- Cline’s multi-file workflows are 39% faster .
- OpenAI models in Cursor became up to 30% faster .
WebSocket mode is the one of the most significant new capabilities in the Responses API since its launch in March 2025. We went from idea to running in production in just a few weeks through close collaboration between OpenAI's API and Codex teams. It not only dramatically improves agent rollout latency but also supports a growing need for builders: as model inference gets faster, the services and systems that surround inference also need to speed up to transfer these gains to users.
Authors
Brian Yu, Ashwin Nathan
Acknowledgements
Special thanks to the Responses API and Codex teams, who worked on creating WebSocket mode.