Developer checked 101 AI agent "tests pass" claims: 35% were false

A developer audited two weeks of Claude Code and OpenAI Codex sessions: of 101 claims that tests, builds or lint passed, 35 were not true at the moment they were made. Almost all false claims were stale — tests had passed earlier, but the agent kept editing code without re-running them.

• 35 of 101 success claims were false at the moment they were made
• 43% of Codex claims and 18% of Claude Code claims were untrue
• Most false claims were stale: tests passed before later code edits
• A detector tool reached only 73% accuracy and was never shipped