Tokenization Inequality
Before an AI model reads your words, it breaks them into pieces called tokens. Because today's tokenizers are optimized around English, the same sentence often requires many more tokens in other languages.
That difference affects cost, context length, and often the quality of the response. Two people asking the same question in different languages may not receive the same value from the same AI model.
The demo below compares how the same sentence is tokenized across languages.
Same sentence, Different Price
See how the same sentence is tokenized across different languages. Counts use OpenAI's GPT tokenizer. Other models vary, but the pattern remains.
- English: Hello, how are you today?7 tokens
- Japanese: こんにちは、今日はお元気ですか？9 tokens
- Spanish: Hola, ¿cómo estás hoy?7 tokens
Mapping tokenization
What better looks like, one waypoint at a time.
Build tokenizers on genuinely multilingual data so no language inherits the leftovers.
Price by meaning delivered, not tokens consumed, so two people asking the same question pay the same.
Report token efficiency across languages as a published benchmark, the way models report accuracy.
Explore byte-level and character-level approaches that remove the vocabulary bottleneck entirely.
Mapped.