OpenAI shows how benchmark tests distort grading of AI-written code

OpenAI detailed flaws in SWE-bench: 68.3% of tasks were dropped during cleanup, and 59.4% of audited SWE-bench Verified problems had issues in tests or problem descriptions. In February 2026 the company stopped reporting Verified scores and now recommends SWE-bench Pro.

• 38.3% of SWE-bench tasks were flagged as underspecified, 61.1% for unfair tests
• After cleanup, GPT-4o's score rose from 16% to 33.2% on 500 tasks
• 59.4% of 138 audited Verified problems had test or description defects
• On SWE-Lancer, Claude 3.5 Sonnet earned $403K of $1M, solving 21.1% of e2e-graded tasks