Study: AI models charge far more tokens for underrepresented languages

The Coalition for Language Equity in AI Research found that LLM tokenizers use 1.5 to 5 times more tokens for less common languages than for English among the 20 languages with the largest web presence. Across 200 languages the gap reached 17 times in one model, raising costs, slowing responses and shrinking usable context windows.

• Among top-20 web languages, rarer ones need 1.5–5x more tokens than English
• Across 200 languages the gap reached 17x in one of the models tested
• Affected groups include Punjabi (~200M speakers), Bengali (~266M) and Urdu (~288M)
• More tokens mean higher bills, slower responses and less room in context windows