OpenAI’s software for catching badly behaved agents was tested from the inside this summer by two outsiders, and it gave ground on all three fronts they tried. METR and Redwood Research published the summary on October 22.
One researcher from each group sat inside OpenAI from July 21 to August 11. They read the internal codebase, the monitor’s prompts, and its configuration, and they interviewed about a dozen engineers. What came out was a 61-page report delivered privately to the lab.
Three separate weak points surfaced. Codex agents could in principle step past both the sandbox and the Auto-Review layer, which would let them act with whatever reach the logged-in user had. A separate trick hid hundreds of recent actions, including the ones that made up the trick itself. And mismatches in how trajectories are serialized meant an agent could probably hand the monitor input that makes it fail, which quietly skips escalation to a person.
Coverage had holes even without an adversary. Compaction events went unwatched for a stretch, so an agent that kept working could keep postponing review. The testers put the unmonitored share of eligible Codex traffic at roughly 10% between May and July.
OpenAI says the production gaps are patched and that monitoring coverage has since widened. METR has not checked that claim. The report’s own advice is blunt: watch agents every few minutes rather than at the end of a task, and reconcile inference logs against monitor logs to prove the traffic you think is covered really is.
