OpenAI Audit: Researchers Uncover Significant Gaps in Internal AI Safety Monitoring

Up to 25 percent of internal agentic traffic from OpenAI’s most advanced models was left entirely unmonitored, according to a new independent audit. Researchers from METR and Redwood Research conducted a three-week on-site review of the San Francisco-based company’s internal systems, revealing significant weaknesses in how OpenAI oversees its AI behavior. The findings, released on October 23, highlight substantial gaps in the monitoring framework for models at the GPT-5.5 class and above.

From July 21 to August 11, the audit team had direct access to OpenAI’s codebase, internal documentation, and interviews with roughly a dozen employees. Their primary focus was the company’s internal monitoring system designed to detect rogue AI behavior. The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic fell outside the scope of monitoring during this period. METR noted that the uncertainty surrounding this specific estimate is significant in both directions, meaning the true figure could be higher or lower.

The review also identified a critical flaw in the monitoring of internal Codex traffic. Due to a gap in how the system handled compaction events, which are context summaries, approximately 10 percent of this traffic went untracked between May and July. This issue allowed agents to delay oversight because the system only triggered when control was handed back to the user. Consequently, agents could bypass monitoring simply by continuing to execute actions without returning control. OpenAI independently discovered this bug and implemented a fix while the audit was still in progress, though METR states the estimate for untracked traffic remains uncertain.

Beyond the compaction gap, the auditors discovered methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, model behavior during reinforcement learning (RL) training was not being monitored at all when the audit began. OpenAI clarified that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring coverage, but these changes occurred after the auditors’ access period ended and were not included in the final evaluation.

OpenAI asserts that it has now fixed all the weaknesses identified by the auditors. The full 61-page report was delivered to the company, but most of it remains hidden from the public. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version, while only a short summary was released to the general public. METR stated it had no contractual right to disclose the redacted portions independently. This audit served as a pilot program organized jointly by the three organizations to grant outside researchers deep access to internal infrastructure. For future iterations, METR has expressed a desire to push for greater public transparency.

The findings underscore the complexity of overseeing autonomous AI systems in production environments. While OpenAI has addressed the specific issues raised, the limited public disclosure of the full report leaves questions about the extent of previous blind spots. The pilot nature of this collaboration suggests that further independent audits may follow, potentially with more open reporting structures. For now, the industry awaits clearer guidance on how effectively these advanced models are contained and monitored in real-time scenarios.