95% recall, 80% answers: LongMemEval retrieval scores mislead
A developer benchmarked two AI-agent memory layers on the same 500 LongMemEval questions: agentmemory's 95.2% recall_any@5 translated to 80.4% answer accuracy, while Graph-MIND reached 89.0% using ~3.4k tokens versus ~16.5k. The gap is statistically significant (exact McNemar p = 3.3e-6).
- agentmemory: 95.2% R@5 but 80.4% correct answers on 499 questions
- Graph-MIND: 89.0% accuracy with ~3.4k tokens vs ~16.5k
- Multi-session questions show the widest gap at 15 points
- Graph-MIND alone was right on 64 questions, agentmemory on 21
Read next
AI