Preprint: detection rate alone ranks the worst LLM code reviewers first

A preprint tested 16 open LLM code reviewers on 180 Python snippets covering nine classes of cryptographic misuse, with three repeats per snippet (8,640 scored calls). codellama 7B and qwen2.5-coder 0.5B flagged 539–540 of 540 snippets, while gpt-oss 20B led on balanced accuracy at 0.952 with a 93.7% TPR and 3.3% FPR.

• 16 open models from 0.5B to 120B parameters, 8,640 scored calls
• codellama 7B and qwen2.5-coder 0.5B flagged 539–540 of 540 snippets
• gpt-oss 20B: balanced accuracy 0.952, TPR 93.7%, FPR 3.3%
• Size correlates with accuracy (Spearman 0.76), but 120B trailed 20B