Trail of Bits deployed autonomous agents to write and check security patches across open source projects, then did something less common: it went back and audited every change its engineers had merged.
The review covered more than 130 pull requests. Twenty-one of them had introduced a regression. Sixteen were functional, one was a performance hit, and four broke builds or tests. None of the confirmed regressions defeated a security protection, which is a narrower result than it sounds.
The failures shared a shape. A patch would resolve the bug that was reported and disturb behaviour that had been working, in a place the project’s existing tests never looked. A single review pass surfaced 58 candidates and confirmed 21, a precision rate near 36 percent.
Trail of Bits built two plugins to impose discipline on the process. One standardises how a patch is written and requires a live demonstration of the bug before work starts. The other, called regression-hunt, sends the fix to a separate agent asked what else it changed.
In preliminary evaluation the second tool produced 25 percent fewer false positives than a baseline Codex agent simply prompted to review pull requests for regressions. The firm is explicit that these are early numbers.
The lesson is about volume. Agent output is a patch proposal, not a patch, and in security-critical code a silent regression is a vulnerability with a delay fuse.
