Practise LLM Fundamentals one question at a time. Answer out loud or in writing, get graded, and see exactly what you missed.
What is a large language model, and how does it actually generate text?
A large language model (LLM) is a neural network trained on huge amounts of text to predict the next token.
The model reads the text so far, scores every vocabulary entry, and samples one token from the resulting probabilities.
tokens = encode(prompt)
while not done(tokens):
logits = model(tokens) # one score per vocabulary entry
probs = softmax(logits[-1]) # next-token probabilities
tokens.append(sample(probs)) # pick one, feed it back in
print(decode(tokens))
Autoregressive means each output becomes part of the next input.
The weights hold patterns, not a fact database, so fluent text is not proof of correctness.
Missed:
“So a token is the unit the model actually reads and writes. Usually it's a common word or a chunk of a word - 'unbelievable' might come out as 'un', 'believ', 'able' - and the model has a fixed vocabulary of them, something like a hundred thousand. Why not whole words? You'd never have a complete list. New words, names, typos, code identifiers - anything that isn't in the list would just be unknown to the model. Why not single characters, then? That would cover everything, but every sentence gets several times longer, and the model's work grows with sequence length, so you'd burn a lot more context and compute. Subword tokens are the middle ground: common words stay one token, and rare stuff breaks into pieces the model already has.”
Knew itCorrectly explains subword tokenization and why fixed word lists fail (OOV) and characters are too costly. Would be stronger with a concrete note on vocabulary size scaling to cover any input.
Missed: Doesn't mention that the ~100k vocabulary can encode any input by falling back to smaller pieces or bytes
Get your own answers graded
Sign up free. Your account comes with 50 credits for grading, and it remembers what you missed.
What it covers
All 58 questions
How LLMs and transformers work
Next-token prediction, the transformer block, self-attention and its cost, position encoding, model variants and mixture-of-experts.
- What is a large language model, and how does it actually generate text?
- Describe the transformer architecture at the level an application engineer should understand. Pro
- Explain self-attention conceptually. What problem does it solve? Pro
- What is the difference between encoder-only, decoder-only, and encoder-decoder transformer models, and where is each used? Pro
- Why is attention quadratic in sequence length, and what does that imply for long-context work? Pro
- What is a Mixture-of-Experts architecture, and why has it become standard for large models? Pro
- How do transformers represent token order, and what are rotary position embeddings (RoPE)? Pro
Tokens and tokenization
What tokens are, how BPE builds a vocabulary, why token counts drive cost and limits, language differences and counting tokens accurately.
- What is a token, and why do models work with tokens instead of whole words or characters?
- Why do token counts matter so much in practice?
- How does byte-pair encoding (BPE) tokenization work? Pro
- Why does the same content cost different amounts in different languages, and why should engineers care? Pro
- How do you count tokens accurately before sending a request, and why not just estimate? Pro
Context windows and caching
What the context window holds, current sizes, quality loss as it fills, measuring effective context and structuring prompts for caching.
- What is a context window?
- What happens to model quality as the context window fills up? Pro
- Compare context window sizes across the major 2026 models. What caveats apply? Pro
- What is prompt caching, and how do you structure prompts to exploit it? Pro
- Advertised context windows keep growing. How do you evaluate what a model can actually use? Pro
Sampling and decoding controls
Temperature, top-p, top-k, stop sequences, why temperature 0 is not deterministic, and other sampling methods and how they fail.
- What does the temperature parameter control?
- How do temperature and top-p interact, and how should you set them in practice? Pro
- What is top-k sampling, and how does it compare with top-p? Pro
- What are stop sequences, and when would you use them? Pro
- Is temperature 0 deterministic? Explain why or why not. Pro
- Discuss sampling beyond temperature and top-p: failure modes and current practice. Pro
Training and alignment
Pretraining, scaling laws and perplexity, base versus instruct models, RLHF, DPO, RLAIF and their failure modes, and LoRA fine-tuning.
- What is the difference between pretraining and post-training?
- Explain RLHF at a high level. Why is it needed? Pro
- What is the difference between a base model and an instruct or chat model, and when does it matter? Pro
- What are scaling laws, and how do they inform model size, data, and compute decisions? Pro
- What is LoRA, and how does parameter-efficient fine-tuning compare with full fine-tuning? Pro
- What is perplexity, and what does it tell you about a language model? Pro
- Compare DPO with PPO-based RLHF for preference tuning. Pro
- What are RLAIF and Constitutional AI? Pro
- What are the known failure modes of RLHF-style post-training? Pro
Hallucinations and knowledge limits
Why models hallucinate, knowledge cutoffs, causes and mitigations, measuring groundedness in production, and using logprobs as a confidence signal.
- What is a hallucination, and why do LLMs hallucinate?
- What is a knowledge cutoff, and how should it influence how you build with a model? Pro
- What practical techniques reduce hallucinations in an LLM application? Pro
- Give a taxonomy of hallucination causes and match mitigations to each. Pro
- How would you measure and monitor hallucinations in a production LLM system? Pro
- Can token log probabilities be used to estimate model confidence, and what are the limits? Pro
Model landscape and multimodality
The 2026 model families, open-weight versus API models, when small models win, and how multimodal models handle images and what they cost.
- What does it mean for a model to be multimodal? Pro
- What does the major model landscape look like in 2026? Pro
- What is the difference between open-weight and proprietary API models, and when would you choose each? Pro
- When does a small model beat a frontier model? Pro
- How do multimodal models process images, and what does that cost in tokens? Pro
Embeddings and retrieval
What embeddings are, how they power semantic search and RAG, choosing an embedding model, and when to pick RAG, long context or fine-tuning.
Reasoning models
What reasoning models are and when they help, test-time compute as a scaling axis, controlling thinking effort and trusting the visible chain of thought.
Inference performance and cost
The KV cache and shrinking it with shared key-value heads, quantization, speculative decoding, reducing latency, model routing and cascading, and the economics of high-volume LLM features.
- What is the KV cache, and why does it dominate LLM inference economics? Pro
- What are the tradeoffs when quantizing a self-hosted model? Pro
- How would you design model routing or cascading to balance cost and quality? Pro
- What techniques reduce LLM inference latency, and which matter most? Pro
- Walk through the economics of a high-volume LLM feature and the levers for optimizing cost. Pro
- What is speculative decoding, and why doesn't it change the model's output? Pro
- What are multi-query and grouped-query attention, and what do they trade away? Pro
All questions
Showing of 58What is a large language model, and how does it actually generate text?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
A large language model (LLM) is a neural network trained on huge amounts of text to predict the next token.
The model reads the text so far, scores every vocabulary entry, and samples one token from the resulting probabilities.
tokens = encode(prompt)
while not done(tokens):
logits = model(tokens) # one score per vocabulary entry
probs = softmax(logits[-1]) # next-token probabilities
tokens.append(sample(probs)) # pick one, feed it back in
print(decode(tokens))
Autoregressive means each output becomes part of the next input.
The weights hold patterns, not a fact database, so fluent text is not proof of correctness.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What is a token, and why do models work with tokens instead of whole words or characters?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
A token is the unit of text a model reads and writes: usually a common word or a word piece. Models have a fixed vocabulary of tokens.
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # OpenAI GPT-4o-family encoding
ids = enc.encode("tokenization is unavoidable")
[enc.decode([i]) for i in ids]
# illustrative split: ['token', 'ization', ' is', ' unavoid', 'able'] -> 5 tokens, 3 words
Whole words fail because new words keep appearing, and any fixed word list would mark them unknown. Characters work, but sequences get much longer, so attention cost grows fast.
Tokens sit in between. A vocabulary of roughly 100k tokens covers any input, because rare words split into known pieces, down to single bytes if needed.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
Why do token counts matter so much in practice?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
Tokens are the unit a model bills, limits and produces. So the token count of a request sets its cost, whether it fits, and how long it takes.
- Cost. Providers charge per token, and output tokens usually cost several times more than input tokens.
- Fit. The prompt plus the output must fit the context window, or the call errors or gets truncated.
- Latency. The model produces output tokens one at a time, so time to finish tracks output length.
A small prompt change can double the bill across millions of calls. So trim old chat history, and set a real max_tokens on every request.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What is a context window?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
The context window is the most tokens a model can handle in one request. The window covers the system prompt, the conversation history, and the tokens the model generates.
A 200K window with a 190K prompt leaves about 10K for the reply, so max_tokens must fit that leftover space.
WINDOW = 200_000
prompt_tokens = count_tokens(messages) # measure it, do not guess
max_tokens = min(4096, WINDOW - prompt_tokens - 500) # keep a safety margin
Models are stateless and keep nothing between calls, so each request resends the whole history.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What does the temperature parameter control?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
Temperature controls how random the next-token pick is. It divides the logits, the raw scores the model gives each token, before softmax turns them into probabilities.
probs = softmax(logits / temperature)
A low temperature sharpens the distribution, so the top token wins almost every time. A high temperature flattens it, so unlikely tokens get picked more often. Typical settings fall into three bands:
0to0.3is near-greedy and more repeatable. Use it for extraction, classification and code or tool calls.0.7to1.0is balanced, for chat, drafting and summaries.- Above
1.2is very flat, with a risk of incoherent text. Use it for brainstorming or generating diverse candidates.
At exactly 0, implementations skip the division and pick the top token. Raising temperature to escape a wrong answer usually just yields a differently wrong one.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What is the difference between pretraining and post-training?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
Pretraining builds general knowledge, and post-training teaches the model how to behave. In pretraining, the model learns to predict missing or next tokens across a huge dataset. The result is a base model that may only continue text, not answer requests.
Post-training then shapes that base model in stages:
- Supervised fine-tuning (SFT) trains on written examples of good answers.
- Preference optimization, through RLHF or DPO, then trains on ranked answer pairs that people prefer.
Pretraining takes far more data and compute than post-training. But post-training can add biases like excessive agreement or refusal.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What is a hallucination, and why do LLMs hallucinate?
Answer it yourself first - out loud, or typed below.
How should your speech become text?
Listening… your words appear above as you speak - tap Stop when you're done.
Recording · cr - tap Stop & transcribe when you're done.
Transcribing with AI…
Voice:
Last attempt -
A hallucination is model output that sounds confident and fluent but is false or unsupported by any source: invented citations, fake API methods, wrong dates.
The model is trained to produce likely text, not verified facts. Knowledge sits compressed in the weights, so gaps get filled with something plausible.
Post-training rewards confident answers, so guessing often scores better than admitting ignorance.
Nothing in the API flags a hallucination. Grounding and verification stay the application's job.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
What is a knowledge cutoff, and how should it influence how you build with a model?
What does it mean for a model to be multimodal?
What are embeddings, in simple terms?
What does the major model landscape look like in 2026?
What is the difference between open-weight and proprietary API models, and when would you choose each?
Describe the transformer architecture at the level an application engineer should understand.
Explain self-attention conceptually. What problem does it solve?
How does byte-pair encoding (BPE) tokenization work?
Why does the same content cost different amounts in different languages, and why should engineers care?
How do you count tokens accurately before sending a request, and why not just estimate?
How do temperature and top-p interact, and how should you set them in practice?
What is top-k sampling, and how does it compare with top-p?
What are stop sequences, and when would you use them?
What happens to model quality as the context window fills up?
Compare context window sizes across the major 2026 models. What caveats apply?
What is prompt caching, and how do you structure prompts to exploit it?
Explain RLHF at a high level. Why is it needed?
What is the difference between a base model and an instruct or chat model, and when does it matter?
What are reasoning models, and when do they actually help?
What is test-time compute, and why did it become a major scaling axis?
What practical techniques reduce hallucinations in an LLM application?
How do embeddings power semantic search and RAG?
How do you choose an embedding model in 2026?
What are scaling laws, and how do they inform model size, data, and compute decisions?
What is the difference between encoder-only, decoder-only, and encoder-decoder transformer models, and where is each used?
What is LoRA, and how does parameter-efficient fine-tuning compare with full fine-tuning?
What is perplexity, and what does it tell you about a language model?
Why is attention quadratic in sequence length, and what does that imply for long-context work?
What is the KV cache, and why does it dominate LLM inference economics?
What is a Mixture-of-Experts architecture, and why has it become standard for large models?
What are the tradeoffs when quantizing a self-hosted model?
Is temperature 0 deterministic? Explain why or why not.
Give a taxonomy of hallucination causes and match mitigations to each.
How would you design model routing or cascading to balance cost and quality?
When does a small model beat a frontier model?
How do you control how much a reasoning model thinks, and why does it matter?
Compare DPO with PPO-based RLHF for preference tuning.
What are RLAIF and Constitutional AI?
How do multimodal models process images, and what does that cost in tokens?
Advertised context windows keep growing. How do you evaluate what a model can actually use?
What techniques reduce LLM inference latency, and which matter most?
How would you measure and monitor hallucinations in a production LLM system?
Walk through the economics of a high-volume LLM feature and the levers for optimizing cost.
Discuss sampling beyond temperature and top-p: failure modes and current practice.
What are the known failure modes of RLHF-style post-training?
When do you choose RAG, long context, or fine-tuning, and how do they combine?
How much should you trust a reasoning model's visible chain of thought?
Can token log probabilities be used to estimate model confidence, and what are the limits?
How do transformers represent token order, and what are rotary position embeddings (RoPE)?
What is speculative decoding, and why doesn't it change the model's output?
What are multi-query and grouped-query attention, and what do they trade away?
This answer is part of Pro.
The full written answer, with the trade-offs and follow-ups an interviewer will probe.
No matches
Try a different filter or search term.
From the blog
Posts about LLM Fundamentals, each written from a video on our YouTube channel, with its charts, code and sources.
LLM Fundamentals in pictures.
A quick recap of LLM Fundamentals in 38 short animated pictures, one idea at a time. About 9 min to get the whole topic back in your head before you walk in. Pause any picture, slow it down, or step through it.
Pro The first chapter is free to play. The other 33 pictures come with Pro.
LLM Fundamentals cheatsheet
- Tokens & generation01
- Context window & caching02
- Sampling & stopping03
- Hallucinations & grounding04
- Embeddings & RAG05
- Inside the transformer06
- Training & fine-tuning07
- Reasoning models08
- Choose a model09
- Speed & self-hosting10
- The model11
- + 5 more inside
This picture is part of Pro. The first chapter is free to play; the other 33 pictures come with Pro.
Get Pro · $10/mo Already Pro? Sign in51 of 58 LLM Fundamentals answers are in Pro.
Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.
- Full answers + code
- AI explanations, simpler or deeper
- 1,000 AI credits / month
- Cancel anytime
Change topic
Pick a different technology or stack. Your current topic stays put until you choose a new one.
MEAN
MongoDB, Express, Angular, Node.jsMERN
MongoDB, Express, React, Node.jsDjango
Python Full-Stack DevelopmentRuby on Rails
Convention over ConfigurationServerless on AWS
Serverless Architecture on AWSInterviewers also test these - they're common to every stack, whichever one you picked above.
Flutter Mobile
Flutter Cross-Platform Mobile DevelopmentInterviewers also test these - they're common to every stack, whichever one you picked above.
Spring Boot
Enterprise Java Development.NET
Microsoft EcosystemVue
Vue.js, Vite, TypeScript, Tailwind, Node.jsGo Backend
Golang, gRPC, PostgreSQL, Redis, RabbitMQInterviewers also test these - they're common to every stack, whichever one you picked above.
FastAPI
Python, FastAPI, SQLAlchemy, PostgreSQLReact Native
React, TypeScript, Redux, FirebaseiOS Native
Swift, SwiftUI, UIKit, FirebaseAndroid Native
Java, Jetpack Compose, FirebaseDevOps / Platform
Docker, Kubernetes, Terraform, CI/CDInterviewers also test these - they're common to every stack, whichever one you picked above.
AI Engineer
LLMs, RAG, Agents, EvalsAI-Powered Developer
Claude Code, Copilot, Agentic WorkflowsCore SWE Interview Prep
Data structures, algorithms, OS, concurrency, networking, gitInterviewers also test these - they're common to every stack, whichever one you picked above.
Interviewers also test these - they're common to every stack, whichever one you picked above.