Click any code to copy it
Done 0/0
To review 0
The model
- An LLM predicts the next token from the tokens so far. It samples one, appends it and runs again, so every output token costs one more pass.
- Tokens are the unit of cost, context limits and latency. Count them with the model's own tokenizer, not by characters.
- The model is stateless. It sees only this request's context window, so you resend the history on every call.
- Knowledge lives in the weights as patterns, not as a fact database. Pass in dates, prices and other changing facts through retrieval or tools.
- Pretraining makes a base model that continues text. Post-training turns it into an assistant, and can teach it to agree too readily.
max_tokens or end of turn.Tokens & generation
how a model reads and writes textHow an LLM writes text
Say what an LLM is trained to doPredict the next token, across huge amounts of text.
Define autoregressiveThe model feeds each output token back in as input.
TrapFluent text isn't proof it's right. The weights hold patterns, not a fact database.
Tokens and BPE
l o w e rstart from single bytesl o w ermerge e + rlo w ermerge l + olow ermerge lo + w
Define a tokenA common word or word piece from the model's fixed vocabulary.
Say why models don't use words or charactersA word list misses new words. Characters make sequences far longer.
Say how BPE builds its vocabularyMerge the most frequent adjacent pair, again and again, to a set size.
TipBytes are the base units, so no input is ever unknown. A bigger vocabulary shortens sequences but grows the embedding matrix.
Count tokens
Say why token counts matterCost, context fit and latency. Output tokens cost several times more.
Count tokens for an OpenAI modelLocally, with tiktoken. No network call needed.
Count tokens for a Claude modelCall count_tokens on the API. The tokenizer isn't public.
Budget tokens for other languagesChinese takes up to 2.5 times the tokens of English, Hindi up to 4.
TrapDividing characters by 4 is a rough guess. It can be off by 2 to 3 times on code and non-Latin text.
Images cost tokens too
Define a multimodal modelIt handles more than one kind of data. Most take images in, text out.
Say how a model reads an imageA vision encoder embeds patches, and a projector maps them to tokens.
Estimate an image's tokens on gpt-4o85 + 170 * tilesAt high detail, per 512px tile of the resized image.
TrapCrossing a 512px tile edge adds a row or column of tiles. At 1040 by 768 an image costs 1105 tokens, at 1024 by 768 only 765.
Context window & caching
what fits in one request, and what the model usesThe context window
Say what the context window holdsSystem prompt, history and the model's reply, in one request.
Leave room for the replyA 190K prompt in a 200K window leaves about 10K for max_tokens.
Keep a conversation goingResend the history. The model keeps nothing between calls.
TrapIf the prompt and the reply don't fit, the call fails or the text gets cut off.
Quality drops as the window fills
Say what "lost in the middle" meansThe model misses facts mid-prompt more than facts near either end.
Name two more ways a long prompt hurtsIrrelevant text pulls attention away. Early rules fade in long chats.
Measure a model's effective contextPlot accuracy by prompt length and fact position.
TipTest with paraphrased facts and multi-hop questions, then set your limit where accuracy falls.
TrapNo error warns you. The model still answers, just less accurately.
Window sizes in 2026
Know the flagship window sizeAbout 1M tokens, like Claude Opus 5.5.
Know smaller and open-weight window sizesAbout 128K to 200K, like Claude Haiku 4.5 at 200K.
TrapThe output cap is separate and much smaller, up to 128K tokens on current flagships.
Prompt caching
Say what a prompt cache hit savesLatency and cost. Cached input costs a tenth of the price or less.
Order a prompt for cachingRules, tools and documents first. History and the question last.
Keep the cache alive longerDefault caches last minutes. Anthropic offers up to an hour, OpenAI a day.
TrapOne changed character breaks the cache for everything after it. Keep timestamps out of the top.
Sampling & stopping
how the next token gets picked, and when output endsTemperature
Say what temperature doessoftmax(logits / temperature)Low values sharpen the odds. High values flatten them.
Set it for extraction or tool calls0 to 0.3, so the output is more repeatable.
Set it for chat or brainstorming0.7 to 1.0 for chat. Above 1.2 to brainstorm, but text can fall apart.
Say why temperature 0 still variesIt's greedy, but batching and MoE routing still shift the logits.
TipFor repeatable output, pin the model version, pass a seed and log system_fingerprint.
TrapRaising temperature to escape a wrong answer usually gives a differently wrong one.
Top-p, top-k and min_p
Say what top-p, or nucleus sampling, keepsThe smallest set of tokens whose total probability reaches p.
Say what top-k keepsThe k most likely tokens, often 20 to 100, however sure the model is.
Choose which sampling knob to tuneTemperature or top-p, not both. Together they're hard to predict.
Say what min_p keepsTokens above a fraction of the top token's probability.
Stop repeats in JSON or codeGrammar-constrained decoding, not repetition penalties.
TrapClaude models since Opus 4.7 reject non-default temperature, top_p and top_k values.
Stop sequences
End output at a markerstop_sequences=["\n\n###"]Handy for splitting records. The reply leaves the marker out.
Tell a deliberate stop from a cutoffresp.stop_reasonIt says stop_sequence, max_tokens or end_turn.
TrapA stop match inside JSON cuts the response, and the parse fails. Use structured outputs to control format.
Hallucinations & grounding
why models make things up, and what stops itWhy models hallucinate
Define a hallucinationFluent, confident output that's false, like a made-up API method.
Say why models hallucinateThey write likely text, not facts. Post-training rewards guessing.
Define a knowledge cutoffThe date its training data ends. The model has no clock either.
Tell the model today's datef"Today is {date.today()}."
TipPin an exact model version. A newer snapshot moves the cutoff and can change answers.
TrapNothing in the API flags a hallucination. Grounding and checking are your application's job.
Ground the answer
Answer only from the CONTEXT below.Quote the sentence you used as [source: doc_id].If the context lacks the answer, reply exactly: NOT_IN_CONTEXT.
Catch claims no source supportsRequire a citation for each claim, and check it in code.
Plan for what still slips throughShow the sources, or hand off to a human.
TrapPrompt polish alone rarely helps. Supply the facts, and let the model say it doesn't know.
Match the fix to the cause
Fix a missing or outdated factRetrieval or tool calls, with citations.
Fix answers built on bad chunksBetter chunking and reranking, plus a grounding check.
Fix sampling noiseLower the temperature, or vote across samples.
TipRetrieval never fixes arithmetic, so use a code tool. Temperature 0 never fixes a false premise.
Measure hallucinations in production
Choose what to measureGroundedness. Check each claim against the retrieved context.
Catch drift after a changeJudge 1 to 5% of live traffic. Check the judge against human labels.
Read confidence from logprobsForce a one-token verdict, read its probability and calibrate it.
TrapA fluent hallucination can still score a high logprob. Logprobs measure likelihood, not truth.
Embeddings & RAG
search by meaning, and when to retrieveEmbeddings
Say what an embedding isA list of numbers for a text's meaning, often 1024 or 1536 long.
Compare two embeddingsCosine similarity. 1.0 means the same direction, 0 means unrelated.
Pick an embedding modelShortlist by MTEB, then measure recall on your own queries.
Choose a hosted or open embedderHosted to start fast. Open weights to keep data in, or at high volume.
TrapEach model has its own vector space. Switching models means re-embedding everything.
Semantic search and RAG
Index documents for searchSplit them into chunks, embed each chunk and store the vectors.
Fetch context for a questionEmbed the query and paste the nearest chunks into the prompt.
Also match exact codes and SKUsAdd BM25 keyword search, then rerank the merged results.
TipRetrieval quality caps answer quality. The model can't use a chunk the search missed.
RAG, long context or fine-tuning
New factsStyleBig corpus
RAGAnswer from a big corpus that keeps changing
Long contextAnswer from a few documents that fit
Fine-tuningFix a format or tone that prompts miss
a good fitdoesn't helpa common mistake
TipThey stack. Prompt first, add RAG when facts are missing, and fine-tune only if evals still fail.
Training & fine-tuning
how a text predictor becomes an assistantPretraining and post-training
Say what pretraining producesA base model that only continues text. It's the costly stage.
Say what post-training addsFine-tuning on good answers, then preference tuning on ranked pairs.
Pick an open-weight model to self-hostThe -Instruct variant, unless you'll fine-tune it yourself.
TrapA self-hosted chat model needs its exact chat template. With the wrong one, it misreads the turns.
RLHF, DPO and RLAIF
Say why RLHF is neededNo one can write a loss for "good answer", but people can compare two.
List RLHF's three stagesFine-tune, train a reward model, then PPO with a drift penalty.
Say how DPO differs from PPOIt skips the reward model and RL loop. Cheaper, but it can't explore.
Say what RLAIF changesA model, not a person, picks the better answer.
Say what Constitutional AI addsWritten principles the model uses to critique and revise answers.
TrapWith AI feedback, the judge model's blind spots and biases pass straight into the policy.
Where RLHF goes wrong
Define reward hackingReward goes up while quality drops. The policy games the reward model.
Say why models flatter and padRaters prefer agreement and long, confident answers.
Name two more side effectsMode collapse cuts variety. The alignment tax weakens base skills.
Scaling laws and perplexity
Say what scaling laws predictLoss falls as a power law in parameters, data and compute.
Split a budget, Chinchilla styleAbout 20 tokens per parameter. Overtrain a smaller model to serve cheaply.
Compute perplexitymath.exp(sum(nll) / len(nll))Lower is better. Compare only models with the same tokenizer.
TrapBoth track loss, not task skill. A lower perplexity doesn't promise better answers.
LoRA and full fine-tuning
Say how LoRA worksh = x @ W + (x @ A @ B) * (alpha / r)Only A and B train. W stays frozen.
Say where LoRA saves memoryNo gradients or optimizer state for the frozen weights.
Serve many fine-tunes cheaplyRun many megabyte adapters over one base model.
TipStart with LoRA. Move to full fine-tuning only when quality stops improving.
Reasoning models
when extra thinking pays off, and what it costsReasoning models and test-time compute
Say what a reasoning model isOne post-trained to write a long chain of thought before it answers.
Pick tasks where reasoning helpsMath, proofs, code and planning. Not extraction or classification.
Define test-time computeCompute spent per request at inference, like thinking tokens.
Say why test-time compute took offPretraining gains cost exponentially more. Thinking longer lifts math and code.
TrapThinking can't add facts the model never learned. Those need retrieval.
Control how much it thinks
Name each API's effort settingreasoning.effort on OpenAI, output_config.effort on Anthropic, thinking_level on Gemini.
Cap thinking tokens on older Claude modelsthinking={"type": "enabled", "budget_tokens": 4000}Keep it below max_tokens.
Say how thinking tokens are billedAs output tokens, and they count toward max_tokens.
TrapToo small a max_tokens can cut the answer off after the thinking.
How far to trust the reasoning
Say what a visible chain of thought isA hint, not an audit trail. It may only be a summary.
Say what a reasoning trace is good forDebugging and eval analysis. Never proof, or an explanation for users.
TrapIn Anthropic's tests, Claude 3.7 Sonnet mentioned a hint it had used only 25% of the time.
Choose a model
hosted or open, small or frontier, and the billModel families in 2026
Name the main hosted model familiesGPT, Claude and Gemini, from OpenAI, Anthropic and Google.
Name the main open-weight familiesLlama, Qwen and DeepSeek. OpenAI also ships gpt-oss.
Choose a hosted API or open weightsAn API for speed and top quality. Open weights keep data in-house.
TrapVendors retire old API model versions on a schedule, and calls to a retired model fail.
Small model, frontier or a cascade
Know when a small model winsNarrow tasks like classification, ideally fine-tuned on your labels.
Know when a frontier model winsOpen-ended reasoning, long agent loops, or no labels to train on.
Cascade from cheap to expensiveRun the small model first, and escalate when a quality check fails.
Estimate a cascade's costc_small + p * c_largep is the share of requests that escalate.
TrapSmall models degrade faster as context fills and across multi-step tool use.
Cut the cost
Model a feature's cost(in_tok * p_in + out_tok * p_out) * callsPer successful task, not per call.
Cut the output billAsk for shorter answers. A max_tokens cap alone cuts them off.
Cut the price of work that can waitUse a batch API. It's half price on Anthropic, with slower turnaround.
TipAttack output tokens and repeated input first. Both scale directly with traffic.
Speed & self-hosting
faster tokens, and models that fit on your GPUsCut the latency
Prefill reads the prompt before the first token, and a cache hit skips most of it. Each dark bar is one output token, which caching can't speed up.
Cut the wait the user feelsStream the tokens. It costs almost nothing.
Cut total time the mostProduce fewer output tokens. The model writes them one at a time.
Cut time to first tokenCache a stable prompt prefix, and send less context.
Cut time per output tokenUse a smaller or quantized model, or speculative decoding.
TrapContinuous batching raises throughput, not latency, and can make p99 worse.
Speculative decoding
Say how speculative decoding worksA small model drafts tokens. The big one checks them in one pass.
Say why the output doesn't changeThe big model accepts each guess with min(1, p / q) and resamples a miss.
TipIt helps most when the draft agrees often, as on code, and less at large batch sizes.
KV cache and grouped-query attention
Say what the KV cache holdsKeys and values of past tokens, at every layer.
Size the KV cache per token2 * n_layers * n_kv_heads * head_dim * dtype_bytes320 KiB per token on Llama 3.1 70B.
Shrink the cache with GQAQuery heads share key-value heads. Llama 3 70B pairs 64 with 8.
Say what MQA trades awayOne key-value head for all. Smallest cache, but quality can drop.
TipThe cache caps batch size, and batch size sets cost per token.
Quantize a self-hosted model
Halve weight memory with little lossFP8 or INT8. It stays close to baseline on most tasks.
Pick the usual sweet spotFour-bit weights like AWQ or GPTQ. Small but real quality loss.
TipDecoding waits on reading weights, so fewer bytes per weight means faster tokens.
TrapPerplexity barely moves while tool-call JSON and long-context recall drop. Test on your own tasks.
Nothing matches. Try a shorter word, or clear the search.
Found this useful? Pass it on.
This sheet stays free. With Pro, you can:
- Every answer on this sheet. Read the full answer to all 58 questions it links, not just the 7 free ones.
- My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.
- PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.
- The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
