LLM Fundamentals.

A large language model is a decoder-only transformer trained on trillions of tokens to predict the next token. Generation is a loop: tokenize the prompt, compute a probability distribution over the vocabulary, sample one token, append it, repeat until a stop condition. Everything else - chat, tool use, agents, reasoning - is scaffolding on top of this loop.

Key consequences of the mental model:

• Knowledge lives in the weights. There is no runtime database lookup, so the model can be confidently wrong.
• The API is stateless. "Memory" is you resending history every request.
• Output is sampled, not retrieved. The same prompt can yield different answers.
• Tokens are the unit of everything: cost, limits, latency, rate limits.
• Chat behavior comes from post-training. The pretrained base model just continues text.

Architecture in one paragraph

Tokens become embedding vectors; dozens of identical layers each apply self-attention (communication between positions) then an MLP (per-position computation, holds most parameters and knowledge), with residual connections and normalization; a final head projects to logits over the vocabulary. Causal masking means each token sees only its past.

Self-attention in one paragraph

Each token emits a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("what I carry"). Relevance = query dot key; softmax turns scores into weights; the token's new representation is the weighted mix of values. Multi-head attention runs many such lookups in parallel with specialized heads (syntax, coreference, copying).

Why engineers should care

Mixture-of-Experts (MoE)

Router sends each token to a few expert MLPs out of many. Big total parameters (capability), small active parameters (compute). Examples: DeepSeek V3 (671B total / 37B active), Llama 4 Maverick (400B / 17B), Mixtral, Qwen3 MoE. Self-hosting rule: judge speed by active params, GPU memory by total params.

How it works

Byte-pair encoding (BPE): start from bytes, repeatedly merge the most frequent adjacent pair, build a vocabulary of ~100K-200K entries. Common words = 1 token; rare words split into subwords; any input is representable. Applied deterministically at inference.

Rules of thumb (English)

• 1 token ~ 4 characters ~ 0.75 words
• 1,000 tokens ~ 750 words ~ 1.5 pages
• Code and JSON: denser in tokens than prose; measure, don't guess
• Non-English text: 1.2x to 4x more tokens depending on language and tokenizer generation
• Images: hundreds to 1,000+ tokens each, resolution-dependent

Counting tokens

# OpenAI: local library
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
n = len(enc.encode(text))

# Anthropic / Gemini: API endpoints
client.messages.count_tokens(model="claude-sonnet-4-5", messages=msgs)
model.count_tokens(contents)  # Gemini

Counts are NOT portable across vendors - each family has its own tokenizer. Chat formatting, tool definitions, and images all add tokens beyond your visible text.

Tokenizer-driven quirks

• Character-level questions fail ("count the r's in strawberry") - the model sees whole tokens.
• Numbers split unpredictably ("12345" may be "123" + "45") - hurts arithmetic; use tools.
• Leading whitespace is part of the token (" the" differs from "the") - matters for stop sequences and prompt assembly.
• Multilingual cost skew: budget per language with real measurements.

Sizes as of early 2026 (verify before relying on)

Output limits are separate and much smaller (commonly 8K-128K). Reasoning/thinking tokens also consume output budget.

Degradation: advertised vs effective context

• "Lost in the middle": retrieval is best at the start and end of context, worst in the middle (U-shaped curve).
• Needle-in-a-haystack is saturated and misleading; harder evals (RULER, NoLiMa, LongBench) show sharp drops - sometimes by 32K tokens - on aggregation and non-lexical retrieval.
• Distractors hurt: more irrelevant text = more wrong-passage answers.
• Long agent transcripts cause instruction drift.

Practical rules:

1. Retrieve, don't stuff. A focused 5K context usually beats a 300K dump.
2. Critical instructions at the start; restate the vital ones at the end.
3. Compact/summarize agent history periodically.
4. Eval at your real operating length (8K vs 64K vs 200K can behave like different models).

Prompt caching

Providers reuse the KV cache for a repeated prefix: big input discounts (cached reads ~10% of price at Anthropic/OpenAI; Gemini has implicit + explicit caches) and much faster TTFT.

Design rules:

• Order prompts stable-first: system prompt, tool definitions, few-shot examples, reference docs, THEN volatile user content.
• One changed byte invalidates everything after it - never inject timestamps or request IDs into the stable prefix.
• Anthropic: explicit cache_control breakpoints; writes cost a small premium, reads are cheap. OpenAI: automatic prefix caching. TTLs are minutes, refreshed on use.
• Biggest win: agents and chat resending growing transcripts every turn.

The dials

Settings by task

Rules of thumb:

• Tune temperature OR top_p, not both.
• Temperature 0 is NOT deterministic (floating-point nondeterminism, batching, MoE routing, fleet heterogeneity). Use seeds as best-effort; design tests around semantic assertions, not string equality.
• Sampling reshapes the model's distribution; it cannot add knowledge or remove systematic error. Temperature 0 still hallucinates.
• Greedy/low-temp decoding can degenerate into repetition loops; penalties or min-p help.

What they are

Models trained with RL on verifiable problems (math, code) to emit a long private chain of thought before answering. Examples: GPT-5.1 with reasoning effort levels, Claude extended thinking, Gemini thinking budgets, DeepSeek R1, Qwen3 hybrid thinking, Mistral Magistral.

Test-time compute

Quality is now a runtime dial: more thinking tokens, self-consistency (sample N answers, majority vote), best-of-N with a verifier, search. Accuracy on hard tasks scales roughly log-linearly with thinking budget.

Controls by vendor

Cost and usage rules

• Thinking tokens bill at OUTPUT rates and can be 10-25x the visible answer. Log them per request type.
• Use for: multi-step math, hard debugging, planning, constraint satisfaction, tricky extraction.
• Skip for: lookups, classification, latency-sensitive chat, high-volume templated work.
• Default low effort; escalate on failure or difficulty; cap budgets on SLA paths.
• Visible chain of thought is not a faithful audit trail (models often don't verbalize what actually drove the answer). Verify outputs, not narratives; don't show raw CoT to users as an explanation.

Causes -> mitigations

Production checklist

• Ground answers in retrieved sources; instruct "answer only from context"
• Permit and test abstention ("if the sources don't say, say so")
• Require citations; programmatically verify cited chunks support claims
• Tools for math, dates, lookups; schema enforcement for structure
• Verification pass (second prompt/model) on high-stakes flows
• Eval set with gold answers + should-abstain cases; track groundedness over time
• LLM-as-judge with rubric; control position/verbosity/self-preference bias; calibrate vs human labels
• Gate model/prompt/retrieval changes behind the eval suite; canary rollouts

Basics

Embedding = vector (256-3,072 dims) where distance encodes meaning. Compare with cosine similarity. Produced by dedicated, cheap models - separate from chat models.

import numpy as np
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

RAG retrieval pipeline

1. Chunk documents (few hundred tokens, respect headings/boundaries, some overlap)
2. Embed chunks; store in vector index (pgvector, HNSW-based stores)
3. Embed query with the SAME model; ANN search top-k
4. Optional: hybrid with BM25 (catches exact IDs/names) + cross-encoder reranker
5. Feed top chunks to the LLM; instruct grounded answering with citations

Evaluate retrieval (recall@k, MRR) separately from generation - most RAG failures are retrieval failures.

Choosing an embedding model (2026)

Selection criteria: retrieval quality on YOUR labeled queries (MTEB is a shortlist tool, not a decision), dimensions vs storage/latency, max input length, multilingual needs, price, license. Switching models later = re-embed the entire corpus, so decide carefully.

Frontier and fast tiers (approximate, early 2026 - always check current docs)

Prices move frequently; long-context and cached tokens have separate rates. Batch APIs: ~50% off everywhere.

Open-weight ecosystem

• Meta Llama 4 (Scout, Maverick - MoE, long context claims), Llama 3.3 still common
• DeepSeek V3.x / R1 - frontier-adjacent quality, MIT license, an order of magnitude cheaper; R1 made open reasoning models real
• Qwen3 - full size range (sub-1B to 235B MoE), hybrid thinking, strong multilingual
• Mistral - Large, Medium, Magistral (reasoning), strong European/enterprise story
• OpenAI gpt-oss, Google Gemma - open offerings from closed labs
• Licenses vary: Apache/MIT (Qwen, DeepSeek, Mistral, gpt-oss) vs conditional community license (Llama)

Knowledge cutoffs

• Training data ends months to ~1.5 years before release; models are unreliable narrators of their own cutoff - check vendor docs.
• Anything time-sensitive (versions, prices, news) must arrive via retrieval or search tools, not weights.
• Pin library versions in prompts ("using React 19") when coding against fast-moving APIs.

The triangle: capability vs cost vs latency

You can usually optimize two. Frontier reasoning = capability at high cost and latency. Nano/Flash class = cost and latency at bounded capability. The craft is routing each request to the cheapest model that clears the quality bar.

Decision heuristics

• Define the eval FIRST, then pick the smallest model that passes it.
• Small models win: classification, routing, extraction from known formats, summarization at scale, autocomplete, voice (TTFT budget), edge/privacy deployments.
• Frontier wins: open-ended reasoning, novel problems, long-horizon agents, messy inputs requiring judgment.
• Fine-tuned small model on a narrow task often matches a prompted frontier model at 10-100x lower cost (distill with frontier-generated data).
• Track cost per successful task, not per token; retries and escalations count.

Routing / cascade pattern

def handle(task):
    tier = difficulty_classifier(task)          # heuristics or tiny model
    out = call(TIER_MODEL[tier], task)
    if not passes_checks(out) and tier != "frontier":
        out = call(TIER_MODEL["frontier"], task)  # escalate once, no loops
    return out

Maintain golden sets per tier; re-tune thresholds whenever any model in the cascade changes.

The pipeline

1. Pretraining: self-supervised next-token prediction on trillions of tokens. Months, huge cost. Produces a base model - knowledgeable text continuer, not an assistant.
2. Supervised fine-tuning (SFT): demonstrations of instruction-following; teaches the chat format and assistant behavior.
3. Preference optimization: RLHF (reward model + PPO with KL anchor) or DPO (direct classification-style loss on preference pairs - simpler, offline, dominant in open source). RLAIF / Constitutional AI replaces human labels with AI feedback guided by written principles.
4. Reasoning RL (RLVR): reinforcement learning with verifiable rewards (math/code checkers); GRPO-style algorithms; produces thinking models.
5. Safety training and specialized tuning (tool use, agentic behavior).

Quick glossary

Known post-training failure modes

Reward hacking (Goodhart on the reward model), sycophancy (flips under pushback; test for it), verbosity bias (longer answers win ratings), calibration loss (uniform confidence post-RLHF), diversity/mode collapse. These are why "the model agreed with me" is not evidence.

• All 2026 frontier models accept images; Gemini adds native audio/video; realtime voice models exist (GPT family); some models generate images/speech.
• Mechanics: vision encoder patches the image and projects into token space (adapter style), or the model is natively trained on interleaved modalities.
• Images are billed as tokens, resolution-dependent: Claude ~ (width x height) / 750 tokens; OpenAI patch/tile-based (~85 to 1,000+); Gemini a few hundred per image/tile.
• Practical: resize to minimum useful resolution, crop to the region of interest, budget vision like text.
• Weak spots: tiny-font OCR, exact counting, precise spatial layout, dense tables - pair with dedicated OCR and validate numbers downstream.

• TTFT is driven by input size (prefill) - shorten prompts, use prompt caching.
• Total time is driven by output tokens - cap max_tokens, request terse formats.
• Stream everything user-facing; perceived latency = TTFT.
• Reasoning effort is pre-answer latency; turn it down where unneeded.
• Parallelize independent calls in pipelines.
• Speculative decoding (draft + verify) gives 2-3x decode speedup with identical output distribution; providers apply it transparently.
• Specialized hardware (Groq, Cerebras) serves open models at extreme token rates; vendors sell priority vs batch tiers.

1. Estimating tokens by character count across languages/vendors - always use the model's own tokenizer.
2. Assuming the context window is memory - the API is stateless; you resend history.
3. Treating temperature 0 as deterministic or as a truthfulness setting - it is neither.
4. Stuffing 300K tokens when 5K retrieved tokens answer better, faster, cheaper.
5. Ignoring prompt structure for caching - volatile data in the system prompt kills the cache.
6. Routing all traffic to the frontier model - most requests are easy; you're paying 10-50x for nothing.
7. Forgetting reasoning tokens bill as output - budgets blow up silently.
8. Trusting a model's claims about its own cutoff, or about current events, without grounding.
9. Prompting "don't hallucinate" instead of grounding + abstention permission + verification.
10. Shipping model/prompt changes without a regression eval - behavior shifts are silent.
11. Using LLM-as-judge without bias controls (position, verbosity, self-preference) or human calibration.
12. Fine-tuning to inject facts - fine-tune for behavior/format; use RAG for knowledge.
13. Wrong chat template on self-hosted open models - quality collapses without errors.
14. Comparing open-weight models by total parameters - MoE speed tracks active params.

• 1 token ~ 4 English chars ~ 0.75 words; 1K tokens ~ 750 words.
• Output tokens cost 4-5x input tokens; thinking tokens are output tokens.
• Cached input ~ 10% of normal price; batch APIs ~ 50% off.
• Effective context < advertised context; test at your real length.
• Retrieval beats stuffing; grounding beats recall; tools beat mental math.
• Tune temperature or top_p, never both; leave reasoning models at defaults.
• Smallest model that clears the eval bar; escalate the residue.
• Cost per successful task > cost per token.
• Verify outputs, not chains of thought.
• Every model upgrade is a behavior change: re-run evals before trusting it.