HKUST: swapping agent harness breaks frozen models
Researchers at HKUST show that benchmarking agent harnesses with a frozen model measures the mismatch penalty, not the harness ceiling. On MiniWoB++, reordering observation fields cut a policy's success rate from 85.0% to 19.2%.
- Return equals harness ceiling minus policy-harness mismatch penalty
- Reordering observation fields dropped success from 85.0% to 19.2%
- DeepAgents ranked first on Terminal-Bench Lite, mini-SWE-agent last
- Fair comparison needs per-harness fine-tuning or a teacher-estimated ceiling
Read next
AI