Picking the longest answer scores 67% on MMLU-Pro biology and psychology questions

An audit by Munim, Inc. found that choosing the longest option without reading the question scores 66.7% on STEM-website biology questions and 67.0% on psychology questions in MMLU-Pro. These questions make up a third of the set and account for 85.8% of the length shortcut's effect.

• Longest-option picking scores 66.7% on STEM-website biology questions
• The same trick scores 67.0% on psychology versus ~11% chance
• STEM questions are a third of MMLU-Pro and 85.8% of the length effect
• Two of six pre-registered predictions failed, including for TheoremQA and SciBench