![\"Diagram](\"https:\/\/blog.quansys.ai\/wp-content\/uploads\/2026\/04\/A_Codex_Agent_Loop_In_Practice.webp\")
<\/figure>\n\n\n\nWhen the API became the bottleneck<\/h2>\n\n\n\n
In the Responses API, previous flagship models like GPT\u20115 and GPT\u20115.2 ran at roughly 65 tokens per second (TPS). For the launch of GPT\u20115.3\u2011Codex\u2011Spark, a fast coding model, our goal was an order of magnitude faster: over 1,000 TPS, enabled by specialized Cerebras hardware optimized for LLM inference. To make sure users could experience the true speed of this new model, we had to reduce API overhead. <\/p>\n\n\n\n
Around November of 2025, we launched a performance sprint on the Responses API, landing many optimizations to the critical-path latency for a single request: <\/p>\n\n\n\n
- \n
- Caching rendered tokens and model configuration in memory to skip expensive tokenization and network calls for multi-turn responses<\/li>\n\n\n\n
- Reducing network hop latency by eliminating calls to intermediate services (for example, image processing resolution) and directly calling the inference service itself<\/li>\n\n\n\n
- Improving our safety stack so we could run certain classifiers to flag conversations faster<\/li>\n<\/ul>\n\n\n\n
With these improvements, we saw close to a 45% improvement in time to first token (TTFT)\u2014which reflects how responsive the API feels\u2014but these improvements were still not fast enough for GPT\u20115.3\u2011Codex\u2011Spark. Even with these improvements, Responses API overhead was too large relative to the speed of the model\u2014that is, users had to wait for the CPUs running our API before they could use the GPUs serving the model.<\/p>\n\n\n\n
The deeper issue was structural: we treated each Codex request as independent, processing conversation state and other reusable context in every follow-up request. Even when most of the conversation hadn’t changed, we still paid for work tied to the full history. As conversations got longer, that repeated processing became more expensive.<\/p>\n\n\n\n
Building a persistent connection<\/h2>\n\n\n\n
To tighten up the design, we rethought the transport protocol: could we keep a persistent connection and cache state, rather than establishing a new connection over HTTP and sending the full conversation history for each follow-up request? The idea was to only send any new information requiring validation and processing and cache reusable state in memory for the lifetime of the connection. This would reduce overhead from redundant work.<\/p>\n\n\n\n
We considered a few different approaches, including WebSockets and gRPC bidirectional streaming. We landed on WebSockets because as a simple message transport protocol, users wouldn’t have to change their Responses API input and output shapes. It was developer-friendly and fit our existing architecture with little disruption.<\/p>\n\n\n\n
The first WebSocket prototype changed what we thought was possible for Responses API latency. An engineer on the Codex team with deep expertise across the API stack pulled together a prototype by running a Codex agent overnight.<\/p>\n\n\n\n
In that prototype, agentic rollouts were modeled as a single long-running Response. Using
asyncio<\/code> features, the Responses API would asynchronously block in the sampling loop after a tool call was sampled, and the Responses API would send aresponse.done<\/code> event back to the client. After executing the tool call, clients would send back aresponse.append<\/code> event with the tool result, which unblocked the sampling loop and let the model continue.<\/p>\n\n\n\nAn analogy here is treating the local tool call as a hosted tool call. When the model calls web search, the inference loop blocks, calls a web search service, and puts the service response in the model context. In our design, we did the same thing; but instead of calling a remote service, we sent the model’s tool call to the client back over the WebSocket. When the client responded, we put the client’s tool call response into the context and continued to sample.<\/p>\n\n\n\n
This design was extremely effective because it eliminated repeated API work across an agent rollout. We could do preinference work once, pause for tool execution, and do postinference work once at the end.<\/p>\n\n\n\n
Unfortunately, this came at the cost of a less familiar and more complicated API shape. We wanted developers to be able to drop in WebSocket support without having to rewrite their API integration around a new interaction mode.<\/p>\n\n\n\n
Keeping the API familiar while making the stack incremental<\/h2>\n\n\n\n
For the version we launched, we switched back to a familiar shape: keep using
response.create<\/code> with the same body, and useprevious_response_id<\/code> to continue the conversation context from the previous response\u2019s state.<\/p>\n\n\n\nOn a WebSocket connection, the server keeps a connection-scoped, in-memory cache of previous response state.<\/p>\n\n\n\n
That cached state includes:<\/p>\n\n\n\n
- \n
- The previous
response<\/code> object<\/li>\n\n\n\n- Prior input and output items<\/li>\n\n\n\n
- Tool definitions and namespaces<\/li>\n\n\n\n
- Reusable sampling artifacts, like previously rendered tokens<\/li>\n<\/ul>\n\n\n\n
![\"Diagram](\"https:\/\/blog.quansys.ai\/wp-content\/uploads\/2026\/04\/From_sequential_requests_to_overlapped_execution__2_.webp\")
<\/figure>\n\n\n\nBy reusing the in-memory previous response state, we were able to land several major optimizations:<\/p>\n\n\n\n
- \n
- Making some of our safety classifiers and request validators process only new input, not the full history every time<\/li>\n\n\n\n
- Keeping an in-memory cache of rendered tokens that we append to so we can skip unnecessary tokenization<\/li>\n\n\n\n
- Reusing our successful model resolution\/routing logic across requests <\/li>\n\n\n\n
- Overlapping non-blocking postinference work like billing with subsequent requests<\/li>\n<\/ul>\n\n\n\n
The goal was to get as close as possible to the minimal-overhead prototype but with an API shape developers already understood and built around.<\/p>\n\n\n\n
Setting a new bar for speed<\/h2>\n\n\n\n
After a two-month sprint building WebSocket mode, we launched an alpha with key coding agent startups so they could integrate it into their infrastructure and safely ramp up traffic. Alpha users loved it, reporting up to 40% improvements\u2060(opens in a new window)<\/a> in their agentic workflows. Given the positive alpha feedback, we were ready to launch.<\/p>\n\n\n\n
The launch results were immediate. Codex quickly ramped up the majority of their Responses API traffic onto WebSocket mode, seeing significant latency improvements. For GPT\u20115.3\u2011Codex\u2011Spark, we hit our 1,000 TPS target and saw bursts up to 4,000 TPS, showing that the Responses API could keep up with much faster inference in real production traffic. The impact showed up quickly in the developer community too:<\/p>\n\n\n\n
- \n
- Codex quickly ramped the majority of their traffic onto WebSockets. Codex users running the latest models such as GPT\u20115.3\u2011Codex\u2060(opens in a new window)<\/a>, GPT\u20115.4\u2060(opens in a new window)<\/a>, and beyond all benefit from WebSocket mode\u2019s speed up.<\/li>\n\n\n\n
- Vercel integrated WebSocket mode into the AI SDK and saw latency decrease by up to 40%\u2060(opens in a new window)<\/a>.<\/li>\n\n\n\n
- Cline\u2019s multi-file workflows are 39% faster\u2060(opens in a new window)<\/a>.<\/li>\n\n\n\n
- OpenAI models in Cursor became up to 30% faster\u2060(opens in a new window)<\/a>.<\/li>\n<\/ul>\n\n\n\n
WebSocket mode is the one of the most significant new capabilities in the Responses API since its launch in March 2025. We went from idea to running in production in just a few weeks through close collaboration between OpenAI’s API and Codex teams. It not only dramatically improves agent rollout latency but also supports a growing need for builders: as model inference gets faster, the services and systems that surround inference also need to speed up to transfer these gains to users. <\/p>\n","protected":false},"excerpt":{"rendered":"
When you ask Codex to fix a bug, it scans through your codebase for relevant files, reads them to build context, makes edits, and runs tests to verify the fix worked. Under the hood, that means dozens of back-and-forth Responses API requests: determine the model\u2019s next action, run a tool on your computer, send the […]<\/p>\n","protected":false},"author":2,"featured_media":69,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-63","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-agents"],"_links":{"self":[{"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/posts\/63","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=63"}],"version-history":[{"count":2,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/posts\/63\/revisions"}],"predecessor-version":[{"id":74,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/posts\/63\/revisions\/74"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=\/wp\/v2\/media\/69"}],"wp:attachment":[{"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=63"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=63"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.quansys.ai\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=63"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Vercel integrated WebSocket mode into the AI SDK and saw latency decrease by up to 40%\u2060(opens in a new window)<\/a>.<\/li>\n\n\n\n
- Codex quickly ramped the majority of their traffic onto WebSockets. Codex users running the latest models such as GPT\u20115.3\u2011Codex\u2060(opens in a new window)<\/a>, GPT\u20115.4\u2060(opens in a new window)<\/a>, and beyond all benefit from WebSocket mode\u2019s speed up.<\/li>\n\n\n\n
- The previous