Audit finds answer leakage in OpenBookQA, SciQ and QASC annotations

MV2 (Munim, Inc.) measured how often the gold answer appears verbatim in the annotations of three science benchmarks. Leakage scales with how answer-shaped the annotation is: 6.36% in OpenBookQA, 91.57% in SciQ and 87.64% in QASC, versus 0.59–4.49% for distractors.

• OpenBookQA: gold in the fact for 6.36% of questions vs 0.59% for distractors
• SciQ: gold in the support paragraph for 91.57% of 12,252 questions
• QASC: gold in combinedfact for 87.64% vs 0.74% for distractors
• In 14.2% of QASC questions the gold appears only in the composed fact