OpenAI Puts AI Agents On ChatGPT Slowdowns And Gets 70% Right

ChatGPT slowdowns: illustration for OpenAI Made A Tool To Hunt Down ChatGPT Slow Replies

OpenAI has handed the job of fixing ChatGPT slowdowns to its own AI agents, and in just over three months they tracked more than 1,700 investigations and pinned down the cause in more than 600 of them.

Key Takeaways

  • Performance Factory is OpenAI’s internal system for running agentic performance engineering workflows
  • Its Regression Workflow opens a task when a performance alert fires, investigates the slowdown, proposes a fix for engineer review, and checks whether performance returned to baseline after deployment
  • OpenAI said that in just over three months the Regression Workflow tracked more than 1,700 investigations and root caused more than 600 cases
  • In a manual review of 100 root cause suggestions, OpenAI graded 70% as accurate

OpenAI said it built the Performance Factory because traditional performance engineering could not keep up with the pace of agentic development. The company described it as an autonomous system that runs in the cloud and automates investigating and resolving performance problems.

The post says the system grew out of several steps, from manual investigations, to Codex-assisted debugging, to shared skills such as perf-forensics, and then to automated workflows that can start from an alert.

How OpenAI Moved Performance Work Into Workflows

Before Codex, OpenAI said engineers manually examined metrics and logs, compared traces, debugged live machines, and read code to investigate slowdowns. Over time, the team built more than 30 performance tools to help with repeated problems.

OpenAI said Codex then became a partner for these investigations. Engineers could ask why something got slower, search codebases, inspect metrics and traces, and follow interactions across services, but they still had to guide the process.

To automate more of that work, OpenAI created a shared skill called perf-forensics. The company said it tells Codex which tools are available, when to use them, and how to work through a slowdown.

OpenAI then moved investigations to the cloud so an alert could start a Codex session automatically. The Performance Factory extends that into end to end workflows that track each task, keep findings with the task, and let work move between agents and engineers.

Also Read: OpenAI Hands TensorWave A 457MW Deal To Run On AMD Chips

What Is Inside The Performance Factory

OpenAI said the system has four parts: a task tracker with configurable workflows, a dispatch loop that assigns ready work to agents, an API that runs Codex in the cloud, and subscriptions that wake agents when something changes.

Engineers define workflows with configuration files and Markdown prompts. OpenAI said the task service tracks states, transitions, and which steps need an engineer, while a dashboard shows open tasks and progress.

The company said its internal API runs Codex turns inside its network with access to code repositories and observability tools such as metrics, logs, traces, CPU profiles, and memory snapshots.

When work needs to wait for something like a review comment, a failed automated code check, or a deployment, the system can subscribe to that event and wake the task when it is ready to continue.

How The Regression Workflow Works

OpenAI said the first workflow inside the Performance Factory is the Regression Workflow. When a configured performance alert fires, it opens a task that moves through stages such as needs investigation, root caused, fix proposed, and resolved.

To find the cause, the agent looks at which metrics got worse, when they changed, and which requests were affected. OpenAI said it also added an adversarial review, a separate Codex session that tries to disprove the proposed diagnosis.

OpenAI gave one example from a ChatGPT investigation. An agent linked failing conversations to image download errors, but a reviewer found that the diagnosis did not explain why failures continued after an internal file location service had recovered, so the investigation was reopened.

Once a diagnosis passes review, Codex can implement a fix and open a pull request for an engineer to review. After deployment, another agent checks whether performance returned to its earlier baseline. If it did, the task is marked resolved.

Performance Factory MetricWhat OpenAI Reported
Time runningJust over three months
Slowdown investigations trackedMore than 1,700
Causes foundMore than 600
Accuracy in a manual check of 100 diagnoses70% graded accurate
Performance tools built before the agentsMore than 30
Source: OpenAI engineering post.

What To Watch Next

The 70% figure cuts both ways. It means the agents got the diagnosis right in most cases OpenAI checked, but also that roughly three in ten were wrong, which is why a second Codex session tries to disprove each diagnosis and an engineer still reviews every fix before it ships.

The bigger shift is AI companies using their own AI to keep their products running. OpenAI said it can turn common optimizations into workflows that run regularly across its services, so the next step is agents that hunt for speedups before users ever notice a slowdown.

Watch for whether OpenAI brings tools like this to Codex customers, who face the same problem of keeping fast-changing software fast.

Read Next: OpenAI’s GPT-6.2 Outsmarts Its Old Self At A Third Of The Cost

Similar Posts