Single agent benchmark runs vary by up to 6 points, study finds

arXiv preprint 2602.07150 found that with the same model, scaffold and temperature, single runs on SWE-bench Verified differ by 2.2–6.0 percentage points. The experiment covered 3 models, 2 scaffolds and 2 temperatures with 10 runs each: 60,000 trajectories and 25.58B tokens.

• DeepSWE-preview on nano-agent: mean 31.4% ± 1.0%, range 28.8% to 32.4%
• Devstral-2 scores 63.5% on nano-agent but 34.9% on R2E-Gym with the same model
• Temperature 0 is not deterministic: standard deviation reaches 1.8 points
• Detecting a 2-point gain needs about 9 runs; a 1-point gain needs 36