Learning path · Models & Tokens · 08
Tokenization
The process of splitting text into subword units (tokens) that the model reads, generates, and bills against.
Why it matters
- Explains why similar-looking strings have different costs and context usage.
- Affects multilingual quality, code handling, and prompt trimming.
- Token boundaries shape what the model can easily memorize or copy.
Key ideas
- Subword units
- BPE and SentencePiece
- Language-dependent density
Models never see raw characters—they see token IDs. English averages roughly four characters per token, but code, URLs, and non-Latin scripts can consume far more tokens per perceived word. Tokenization explains odd behaviours: the model may struggle to spell backwards or count letters because those operations are unnatural at the token level. When engineering prompts, measure tokens, not characters, and trim aggressively near context limits. Build internal token estimators into authoring tools so support and legal teams see budget impact before pasting megabyte PDFs into chat widgets.
Updated 2026-08-09 · Full learning path