Protege launches independent healthcare AI evaluations

Protege has launched a platform for independent evaluation of healthcare AI on patient cases hidden from model developers. Its first benchmark, Disease Odyssey, found leading models identified conditions like sleep apnea and anemia in only 53–69% of cases from patient histories.

• Disease Odyssey: leading models caught sleep apnea and anemia in 53–69% of cases
• Double-blind setup keeps test patient data hidden from model builders
• Xlue AI's HealthSAW beat leading models on 14 of 16 tests at under 0.1% cost
• Standard Model Biomedicine costs 36–170x less per prediction than leaders