Anthropic trial: developers using AI scored 17 points lower on quiz

In Anthropic's randomized trial, 52 engineers learning the Trio library scored 50% on a quiz with an AI assistant versus 67% for those coding by hand. Task time differed little (about 23 vs 25 minutes), and the largest gap was on debugging questions.

• Quiz had 14 questions worth 27 points across seven Trio concepts, 26 participants per arm
• Score gap of 4.15 of 27 points (d=0.738, p=0.010); time difference not significant (p=0.391)
• Median errors during the task: 3 for hand coders vs 1 for AI users
• Conceptual-inquiry group scored about 65%, AI-delegation group about 39%