What is a token, and why do models work with tokens instead of whole words or characters?

beginner

Re-explain

A token is the unit of text a model reads and writes: usually a common word or a word piece. Models have a fixed vocabulary of tokens.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")   # OpenAI GPT-4o-family encoding
ids = enc.encode("tokenization is unavoidable")
[enc.decode([i]) for i in ids]
# illustrative split: ['token', 'ization', ' is', ' unavoid', 'able'] -> 5 tokens, 3 words

Whole words fail because new words keep appearing, and any fixed word list would mark them unknown. Characters work, but sequences get much longer, so attention cost grows fast.

Tokens sit in between. A vocabulary of roughly 100k tokens covers any input, because rare words split into known pieces, down to single bytes if needed.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for LLM Fundamentals One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How does a tokenizer like BPE learn which word pieces go into its vocabulary? Start from bytes, repeatedly merge the most frequent adjacent pair in a training corpus, then replay those merges in order when encoding.
Why do LLMs struggle to count the letters in a word or spell it backwards? The model sees token ids, not characters. It learns what letters are inside a token only indirectly, so character-level tasks are error-prone.
What do you gain and lose by making the vocabulary bigger? Bigger means shorter sequences, but also a larger embedding and output layer, and rare tokens that barely appear in training.
Why can the same text take more tokens in one language than another? The vocabulary is learned mostly from English-heavy data, so other scripts split into more, smaller pieces. Cost, latency and context use all go up.
How would you check a prompt's token count before sending it? Use the tokenizer that matches the model, or the provider's counting endpoint. Rules of thumb drift across languages, code and model families.

What you can say

A token is the unit of text a model reads and writes, usually a common word or a piece of a word, taken from a fixed vocabulary.
For example, 'tokenization is unavoidable' might split into pieces like 'token', 'ization' and 'able', which comes to five tokens for three words.
Whole words don't work because new words keep appearing, and any fixed word list would mark them as unknown.
Characters would cover everything, but sequences get much longer, and attention cost grows fast with length.
Tokens sit in between. A vocabulary of around 100k tokens covers any input, because rare words split into known pieces, down to single bytes if needed.
The downside is that the model sees pieces, not letters, and rare words or less common languages take more tokens, so they use more context and cost more.

Weak answers to avoid

Says a token is just a word Tokens are often word pieces, and one word can be several tokens. Say that plainly, or the interviewer will doubt you can reason about cost and limits.
Can't say why not characters Characters do cover any text. The real problem is length. Sequences get much longer and attention cost grows fast, so say that.
Thinks rare words become unknown tokens Modern byte-level tokenizers split rare words into known pieces, down to single bytes, so no input is ever unknown.
Treats '1 token = 4 characters' as exact That ratio is a rough average for English. Code, numbers and other languages differ a lot, and each model family has its own tokenizer.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the LLM Fundamentals cheatsheet.

From the blog: Jev explained: the AI model that returns decisions, not text

← Back to all LLM Fundamentals questions

Pro · $10/mo

51 of 58 LLM Fundamentals answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →