UserProxyBench: 24.4% of successful agent benchmark runs break the simulated user's own contract

UserProxyBench, a paper by Ashish Jain and Armaan Sandhu, adds an evaluation layer to the tau-bench family to score the simulated user in interactive agent benchmarks. Across 375 tasks and 2,141 criteria in four enterprise domains, swapping the user proxy while holding the agent fixed shifts mean reward by 15.2 points, and 24.4% of successful episodes contain a user-specification violation.

• Swapping the user proxy moves agent reward by 15.2 points with the agent held fixed
• 24.4% of successful episodes contain a user-specification violation
• User Fidelity Score across seven proxies ranges from .599 (GPT-5.4-mini) to .864 (Gemini-3.5-Flash)
• Mean agent turns correlate with UFS at r=.955, while task reward correlates at only r=.594