Audit: picking the longest answer scores 43% on OpenBookQA vs 25% chance

A 51-benchmark audit found that a "pick the longest option" rule scores 43.35% on OpenBookQA against a 25% random baseline, a 1.7x lift with no question understanding. ARC-Easy and ARC-Challenge show no such shortcut, scoring 24.9% and 27.8% at the same baseline.

• OpenBookQA: longest-answer rule scores 43.35% vs 25% random baseline
• ARC-Easy 24.9% and ARC-Challenge 27.8% — essentially at chance
• In ARC the gold is the unique longest option only 20.5–22.4% of the time
• By whitespace tokens the effect vanishes: 17.98% on OpenBookQA