A token is a unit of text that a language model reads or writes. It may be a whole word, part of a word, punctuation, or whitespace. A tokenizer converts text into token IDs before the model processes it.
Tokens balance vocabulary size and sequence length. Character-based input would create very long sequences. Whole-word input would need a huge vocabulary and handle new words, typos, code, and many languages poorly. Subword tokens can represent any text while keeping sequences manageable.
Tokenization matters because context limits, cost, latency, and rate limits are usually measured in tokens. The exact count depends on the model's tokenizer, so estimates based only on characters can be inaccurate.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓