METR and Redwood report OpenAI monitors failed to catch up to 25% of elite model actions
A three-week internal audit revealed that 6 to 25 percent of GPT-5.5 class agentic traffic was unmonitored, while a separate bug left roughly 10 percent of Codex activity untracked.
By Leila Reynolds ·
Researchers from METR and Redwood Research identified major gaps in OpenAI's internal systems for tracking model behavior. The audit found that between 6 and 25 percent of agentic Responses API traffic from models in the GPT-5.5 class and above was not monitored at all during the review period. METR noted that the uncertainty surrounding this range is considerable in both directions.
The assessment took place from July 21 to August 11. It was a joint pilot program that granted the two outside organizations access to OpenAI's codebase, internal documents, and approximately a dozen employees. The goal was to test the company's infrastructure with external oversight.
A separate issue affected internal Codex traffic. METR estimates that roughly 10 percent of this traffic went untracked between May and July. The gap occurred because the monitoring system failed to correctly handle compaction events, which serve as context summaries. OpenAI identified this bug independently and applied a fix while the audit was still ongoing.
The auditors also found that agents could delay oversight. The monitoring system was designed to trigger only when an agent returned control to the user. Consequently, an agent that continued executing actions without handing off could avoid detection. The researchers demonstrated methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox.
Further findings indicated that specially crafted inputs could likely crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning training was not monitored at all. OpenAI stated that it paused reinforcement learning training following an incident in August with Hugging Face. The company later expanded monitoring to cover this area, though these changes happened after the auditors' access period ended.
OpenAI says it has corrected all the weaknesses identified by the researchers. The full report consists of 61 pages and was shared internally. METR and Redwood Research leadership received a redacted 38-page version, while the public received only a short summary. METR stated it did not have the right to disclose the redacted sections independently. The group says it will push for greater public transparency in future audits.
