AI Safety Oversight: External Auditors Identify Significant Monitoring Deficiencies at OpenAI

The ongoing development of autonomous artificial intelligence systems has raised persistent questions regarding how companies monitor the behavior of these models once they are deployed in internal environments. Recent findings from an independent review indicate that substantial gaps existed in the safety monitoring infrastructure at OpenAI during the summer of this year.

Between July 21 and August 11, researchers from METR and Redwood Research conducted a detailed on-site audit of OpenAI’s systems designed to detect rogue or unexpected behavior in AI agents. The team was granted access to the company’s codebase and internal documentation, and they conducted interviews with approximately a dozen employees to assess the robustness of the current monitoring protocols.

The audit process followed a structured timeline that began with preliminary access to internal data and concluded with a comprehensive assessment of system vulnerabilities. This three-week period allowed the auditors to examine how the company’s internal agentic Responses API handled traffic from models at the GPT-5.5 class and above. The researchers focused specifically on identifying instances where automated agents operated without sufficient oversight or logging.

One of the primary findings of the investigation was that a significant share of this high-level traffic was not covered by monitoring at all during the audit window. METR estimates that between six and 25 percent of the relevant traffic lacked any surveillance, although the organization noted that the uncertainty surrounding this specific estimate is considerable in both directions. This gap suggests that a non-trivial number of autonomous actions occurred in a state where they were not being tracked by the company’s safety systems.

Additionally, the auditors identified a specific technical flaw affecting internal Codex traffic. They estimate that roughly 10 percent of monitored traffic went untracked between May and July due to a failure in how the system handled compaction events, which are essentially summaries of context. While OpenAI had independently discovered and fixed this bug while the audit was still in progress, the auditors noted that the system’s design allowed agents to delay oversight by continuing to execute actions rather than handing control back to the user.

The review also uncovered more critical vulnerabilities, including methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox environment. The researchers determined that specially crafted inputs could likely cause the monitor to crash entirely. Furthermore, they found that model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit period, a gap that was addressed later in the year.

OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring to cover this activity. However, the company clarified that these changes were implemented after the auditors' access period had ended, meaning they were not part of the initial evaluation conducted by METR and Redwood Research. The company asserts that all weaknesses identified by the auditors have since been fixed.

Despite the completion of the audit, most of the full 61-page report remains inaccessible to the general public. After OpenAI applied redactions, the leadership of METR and Redwood Research received a 38-page version, while only a short summary was released publicly. METR indicated that it did not have the contractual right to disclose the redacted portions on its own initiative, limiting the amount of technical detail available to outsiders.

This audit was part of a pilot program organized jointly by the three organizations to provide outside researchers with deep access to internal infrastructure. Looking forward, METR has expressed a desire to push for greater public transparency in future audits. As the industry continues to develop more capable AI systems, the balance between proprietary confidentiality and public safety oversight remains a central point of discussion.

The findings highlight the complexities of monitoring autonomous systems that operate within large-scale internal environments. While the company has addressed the specific issues identified, the pilot nature of this audit sets a precedent for how independent researchers may engage with major AI developers in the coming years. The focus will likely shift toward ensuring that these independent checks yield more detailed public reports, allowing for broader scrutiny of AI safety measures.