Study finds AI models cost more in less-spoken languages

The Coalition for Language Equity in AI Research found a hidden "tokenizer tax": among the 20 most popular internet languages, the costliest uses 1.5–5 times more tokens than the cheapest. Expanding to 200 languages widened the gap to 17 times in one model, raising cost and latency and shrinking effective context.

• Among 20 popular languages, token usage varies 1.5–5 times
• Across 200 languages the gap reached 17 times in one model
• Scripts like Burmese, Tibetan and Khmer fare worst
• More tokens mean higher latency and less effective context