Audit: 'pick the longest answer' scores 52.7% on MMLU international law
An MV2/Munim audit found that a question-free 'pick the longest option' heuristic scores 28.05% on MMLU versus 25% chance. The edge is uneven: 23 of 57 subjects show a lift of 5+ points, reaching 52.7% on international law but falling below chance (17.7%) on public relations.
- Pick-longest heuristic scores 28.05% on MMLU vs 25% chance
- International law hits 52.7%, correct options average ~30 chars longer
- 23 of 57 subjects show a lift of 5+ points; 42 of 57 beat chance
- On public relations, pick-shortest is better at 32.3%
Read next
AI