Audit finds 25 defects in LLM benchmark metrics and gold labels

The eval-reliability-audits project published 25 reproducible audits of scoring code and gold labels behind popular LLM benchmarks. It found conflicting labels in SNLI, MultiNLI and ANLI, a 5.4% mismatch between the #### answer and the last annotation in GSM8K, and normalization bugs in exact_match and ROUGE.

• SNLI: 64 exact and 82 normalized conflicting label pairs, all in train
• GSM8K: #### answer differs from last annotation in 470 of 8,679 solutions (5.4%)
• HuggingFace exact_match normalization flips verdicts in both directions
• ROUGE tokenizer can return F=1.0 for summaries with opposite meaning