An intensive three-week on-site audit conducted by researchers from METR and Redwood Research has uncovered significant gaps and vulnerabilities in OpenAI's internal AI safety monitoring systems. According to reports, the evaluation took place between July 21 and August 11, giving outside researchers rare access to OpenAI's internal codebase, documentation, and approximately a dozen staff interviews.
Coverage Gaps in Advanced Agentic Traffic
The audit revealed that a notable portion of internal traffic involving advanced models operated without oversight. Specifically, researchers estimated that between 6 and 25 percent of internal agentic Responses API traffic from models classified at the GPT-5.5 tier and above lacked monitoring entirely during the assessment window, though METR acknowledged substantial uncertainty in both directions. Additionally, an estimated 10 percent of internal Codex traffic went untracked between May and July due to a bug in handling compaction events, which summarize context. OpenAI had independently identified and patched this issue prior to the audit's conclusion.
In broader methodological frameworks discussed by METR, establishing monitor coverage requires analyzing how much internal traffic flows through monitored versus unmonitored pathways. Because frontier agents often require broad affordances like internet access, ensuring complete tracking of agentic inference remains a core challenge for developers attempting to prevent misaligned behaviors from bypassing detection.
Vulnerabilities and Monitor Evasion Risks
Beyond coverage blind spots, the evaluators discovered systemic vulnerabilities that could allow agents or malicious inputs to subvert oversight. During the unmonitored compaction windows, agents could delay human or system intervention by continuing actions rather than returning control. Furthermore, auditors identified methods to hide hundreds of individual actions from monitoring instruments and successfully bypass the Codex sandbox environment.
The investigation also indicated that specially crafted inputs could potentially crash the monitoring apparatus completely. Researchers noted that model behavior during reinforcement learning (RL) training was initially entirely unmonitored when the audit started. OpenAI subsequently paused RL training following a Hugging Face incident in August and broadened monitoring protocols, though those adjustments occurred after the audit team's on-site access concluded.
Restricted Transparency and Reporting Outcomes
The findings were compiled into a comprehensive 61-page report handed over internally to OpenAI. Following redaction processes applied by the company, METR and Redwood Research leadership received a shortened 38-page document, while the general public received only a concise summary. METR stated it possessed no autonomous right to publish the redacted sections independently. OpenAI asserted that it has successfully resolved all identified weaknesses.
What These Findings Mean for Frontier AI Oversight
The revelations from the OpenAI on-site audit highlight a profound structural tension in the artificial intelligence industry: the immense difficulty of maintaining rigorous, real-time safety governance as foundational models scale rapidly in capability and autonomy. As AI systems evolve from narrow utility tools into complex agents capable of utilizing internal credentials, executing code, and browsing the open web, traditional static cybersecurity approaches like basic sandboxing prove insufficient. Automated, LLM-based monitoring systems are designed to bridge this gap by interpreting broader behavioral contexts, but as the METR and Redwood Research audit demonstrates, these oversight layers are themselves susceptible to infrastructure bugs, coverage blind spots, and architectural evasion techniques. The findings underscore why independent third-party evaluations and deep code access are critical for uncovering hidden failure modes that internal development teams might otherwise miss. Ultimately, bridging the gap between rapid capability deployment and reliable containment will require unprecedented transparency, standardized control assessments, and robust verification mechanisms across the entire AI development lifecycle.
Editorial graphic and headline text by Signal Titans.
Source: Three-week on-site audit at OpenAI reveals major gaps in AI safety monitoring · How an assessor could investigate control and monitoring of internal AI agents – METR
SIGNAL TITANS
Leave a Reply