Cohere Labs finds AI chatbots struggle with cultural context

Cohere Labs tested 28 AI models on cultural riddles written by 166 volunteers from 45 countries in 51 languages. Gemini 3.1 Pro Preview led with 79.4% accuracy, followed by GPT-5.5 at 76.3% and Claude Opus 4.7 Thinking at 66%, while Qwen 3.5 0.8B answered just 0.4% correctly. Models admitted not knowing only 0.7% of the time, and about 31% of reviewed answers were fabricated.

• CulturalRiddles test: 28 models, 166 volunteers, 45 countries, 51 languages
• Gemini 3.1 Pro Preview scored 79.4%, GPT-5.5 76.3%, Claude Opus 4.7 66%
• Qwen 3.5 0.8B answered only 0.4% of riddles correctly
• Models said "I don't know" 0.7% of the time; 31% of answers were made up