Token Atlas LLM tokenizer fertility by language

Multilingual token cost for LLMs: how many more tokens a model spends on a language than on English for identical content, measured as tokens per word and tokenizer fertility. More tokens means higher cost, slower output and less text per context window.

Suggest a model or dataset

Where the language is spoken

Selected language, shade by speakers when knownOther countriesZoom: + / − buttons, Ctrl+scroll or two-finger pinch

Tokens per word and tokenizer fertility

Variation across corpora

Multilingual token cost versus mainstream languages

Dashed line = English (1.00×). Bars show tokens relative to English on the same text.

Tokenizer language bias across tokenizers

Language table: tokenizer fertility by language

Suggest a model or a dataset

Missing a model, a tokenizer or a corpus you would like measured? Tell us here. Suggestions are visible to everyone who opens this page, and the status shows what happens next.

  • Model: open weights and a public tokenizer file. Gated repositories are fine if access can be requested.
  • Dataset: the same content in many languages (parallel or aligned), a licence that allows research use, and a direct link.

Suggested so far

  • No suggestions yet.

Method and limits

Models from the ranked list that are not measured