LSCBench audit: 2,700 agent traces recomputed against the leaderboard
LSCBench, a secure-code benchmark from NUS and the Qwen team, ships 2,700 agent trajectories across 450 tasks built from 150 real vulnerabilities in seven languages. Recomputing from catalog.json matched the paper's numbers except Kimi-K3's functional pass rate, 92.2% versus 91.3%.
- Functional completion ranges from 73.6% to 92.2%, but functional-and-secure only 16.4% to 25.1%
- Only Kimi-K3 differs from the paper: 92.2% in the catalog vs 91.3% reported
- 120 of 2,700 runs (4.4%) ended on harness errors, and 15 of them still passed
- The single reward bit conflates a security miss with a build failure
Read next
AI