home
›
cheatsheets
›
ai_engineering
›
local_llms
Free to read Every card and diagram, and Quiz me
2
Free account Star cards. Your Quiz me marks are kept, so next time you drill just what you missed
3
Pro All 55 answers on this sheet, My sheet across every topic, and PDFs
A free account keeps your stars and Quiz me marks. Pro adds all 55 answers on this sheet, My sheet across every topic, and PDFs.
/
Read
Quiz me
Comfortable
Compact
0
Click any code to copy it
Done 0 /0
To review 0
On this sheet
01 Open models & licenses02 Self-host or hosted API03 Run a model locally04 VRAM & KV cache math05 Quantization06 Prompts, sampling & JSON07 Serve many users08 Fine-tunes & adapters09 Scale across GPUs10 Evaluate, ship & monitor
The model An open-weight model ships its weights as a download. Its license, not the word open, says what you may do. Memory is weights plus KV cache plus overhead . The KV cache grows with context and with every user. Decode reads every weight for each token, so memory bandwidth sets its speed and batching is nearly free.Pick the runtime by concurrency . Ollama and llama.cpp serve one user. vLLM and SGLang batch many users. Every runtime speaks the OpenAI-compatible API, so switching is mostly a base URL change. Prefill handles the whole prompt in one pass. Each decode step makes one token and reads all the weights and the cache, so memory bandwidth sets its speed.
Open models & licenses what you can download, and what you may do with it Model families and who makes them Name the US families Llama by Meta, gpt-oss by OpenAI, Gemma by Google, Phi by Microsoft and Olmo by Ai2.
Name the Chinese and French families Qwen by Alibaba, GLM by Z.ai, Kimi by Moonshot AI, and DeepSeek. From France, Mistral.
Open-weight, open source, proprietary Model Data Code
Open-weight, like Llama or Qwen. Get the weights, but not data or code
Open source, like Ai2's Olmo. Get the weights, data and code
Proprietary. The weights never ship. Reach the model only through an API
Trap Most models called open source are only open-weight. Llama's weights are free to download, but its license adds use limits.
Read a model license Find the families under Apache 2.0 or MIT Apache 2.0 for Qwen, gpt-oss, Olmo and Gemma 4. MIT for DeepSeek, GLM and Phi.
Tell Apache 2.0 from MIT Apache adds a patent grant, and you must mark your changes.
List what the Llama license adds An acceptable-use policy, a Built with Llama notice, and a deal above 700M monthly users.
List what the Gemma terms add Use limits you must pass to every recipient, and Google may tell you to stop. Gemma 4 is Apache 2.0.
Trap Custom licenses reach your fine-tunes. A Llama fine-tune's name must start with Llama.
Self-host or hosted API when owning the GPU pays off Self-host or use a provider Keep prompts inside your network Self-host. Air-gapped sites have no other option.
Avoid surprise model changes Pin the weights, so no deprecation or silent update reaches you.
Know when to stay on a hosted API Spiky or low traffic, and the hardest tasks, where frontier models still lead.
Run an open model without buying GPUs Use Together or Fireworks. They sell open models per token.
Get the fastest token generation Groq. It runs custom chips and sells speed, not catalog breadth.
Tip Start hosted. The weights are open, so you can move the same model in-house later without rewriting prompts.
Trap Owned GPUs only save money while they stay busy. An idle GPU bills the same.
Cost per million tokens Self-host dollars per million tokens gpu_hr / (tok_s * 3600) * 1e6 / util $2/hr GPU at 2,500 tok/s = $0.22 full load 0.22 / 0.40 = $0.56 40% busy
Add what the invoice never shows Spare GPUs for failover, reserved commitments, and engineers on call.
Decide when to self-host Large, steady volume, with the model pinned for months.
Trap Measure tokens per second yourself, at your p95 latency target. A vendor's peak number is higher than you'll get.
Run a model locally one user, one machine Ollama, llama.cpp and LM Studio Pull a model and chat with it ollama run llama3.2
Point an OpenAI SDK at Ollama export OPENAI_BASE_URL=http://localhost:11434/v1
Serve a GGUF file with some layers on the GPU llama-server -m model.gguf -ngl 32 --port 8080
Name the hardware llama.cpp runs on CPUs, plus Metal, CUDA, Vulkan or ROCm GPUs. A laptop runs a small model.
Say what LM Studio is A closed-source desktop app over llama.cpp and MLX. Its server listens on port 1234.
Tip Ollama ships each model with its chat template and defaults, so you don't format prompts by hand.
Trap All three are built for one user. llama-server batches requests, but its throughput stays flat as load grows.
GGUF files Say what one GGUF file holds The tensors plus the tokenizer, architecture and chat template.
Read the quant level in a GGUF file name Q4_K_M means a 4-bit k-quant, medium size.
Trap GGUF is for inference only. You convert to it from Hugging Face weights, and you never train in it.
Fix slow local responses Find which phase is slow Read load_duration, prompt_eval_duration and eval_duration in Ollama's response.
Check the model fits in VRAM ollama ps 100% GPU is healthy. A CPU/GPU split is slow.
Halve the KV cache's memory in Ollama OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
Trap 100% CPU on a GPU machine means the GPU lacked free VRAM, or the runtime never found it.
On-device inference Name what limits a phone Memory, then memory bandwidth, then throttling once the chip heats up.
Handle requests too hard for the device Send them to a server, and keep a small model on the device.
Tip You gain privacy, offline use and no per-token bill.
VRAM & KV cache math will it fit, and for how many users VRAM to load a model GB = billions of params x bytes per weight 8B at FP16 = 8 x 2 = 16 GB 16 GB + 20% overhead = about 19 GB
Find the bytes per weight FP16 takes 2, FP8 or INT8 takes 1, and 4-bit Q4_K_M about 0.6.
Check whether a 70B fits one 80 GB card No at FP16, which needs 140 GB. Yes at 4-bit, about 50 GB with overhead.
Trap 4-bit isn't exactly 0.5 bytes. Files also store scales and keep some tensors at higher precision.
The KV cache KV bytes per token 2 x layers x kv_heads x head_dim x bytes 2 x 32 x 8 x 128 x 2 = 128 KiB Llama 3.1 8B, FP16 128 KiB x 128k tokens = 16 GiB one long request
Say what the KV cache stores Keys and values of every past token, per layer, so they aren't recomputed.
Work out how many requests fit Divide the VRAM left after weights and overhead by KV per token. An 8B on an H100 fits about 50 at 8k.
Trap Quantizing the weights doesn't shrink the KV cache. It stays FP16 by default.
Shrink the KV cache Halve it for a small accuracy hit vllm serve <model> --kv-cache-dtype fp8
Cap the context window to fit more requests vllm serve <model> --max-model-len 32768
Know what GQA and MLA save GQA cuts it 4 to 8x, and DeepSeek's MLA about 93%. Both ship with the model.
Trap KV compression fails at long context first. Test quality on your longest prompts.
Dense vs mixture-of-experts Size an MoE's memory By total parameters, since the router can call any expert.
Estimate an MoE's speed By active parameters. A 30B-A3B computes about 3B per token.
Serve an MoE across GPUs vllm serve <moe-model> --enable-expert-parallel --data-parallel-size 8
Tip On one GPU, llama.cpp can keep attention on the GPU and the experts in system RAM.
Pick a model size Classify, extract or route One to eight billion parameters.
Answer over RAG, summarize or draft About eight to thirty billion.
Run agents, multi-step reasoning or hard code Thirty billion and up, or an MoE.
Tip Start one size above your guess, then step down until quality breaks. The last size that passed wins.
Quantization fewer bits per weight, for memory and speed What quantization does, and how far to go Say what quantization does It stores weights in fewer bits. 4-bit is about a quarter of the 16-bit size.
Pick a default level 4-bit, as AWQ or Q4_K_M, unless your eval set shows a real drop.
Spend a fixed VRAM budget A larger model at 4-bit usually beats a smaller one at 8-bit.
Push a quantized model's accuracy further Calibrate on your own traffic. Keep embeddings and lm_head at higher precision.
Trap Quality holds to about 4 bits, then drops fast. Code and math show the damage before chat does.
GGUF, AWQ, GPTQ or FP8 Run on a laptop, a Mac or with CPU offload GGUF k-quants in llama.cpp, Ollama or LM Studio.
Fit a big model in tight GPU memory AWQ, or GPTQ if there's no AWQ checkpoint. Both are 4-bit and close in quality.
Keep near-lossless quality on a newer GPU vllm serve <model> --quantization fp8 FP8. It needs Hopper, Ada or newer.
Tell AWQ from GPTQ GPTQ pushes rounding error into later weights. AWQ protects salient channels first.
Trap FP8 only halves memory. A 70B model still needs two large cards.
Prompts, sampling & JSON get the output you asked for Instruct models and chat templates Pick a base or an instruct model Instruct, unless you'll fine-tune. Look for -instruct or -it in the name.
Define a chat template A Jinja template that turns messages into the exact string the model trained on.
Build the prompt string yourself Call apply_chat_template with add_generation_prompt=True .
Tip Chat endpoints apply the template for you. Raw completion endpoints don't.
Trap Wrong turn markers fail quietly. The model ignores the system prompt, runs past its stop, or breaks tool calls.
Sampling settings Set temperature 0 for extraction, 0.7 for chat. Set it first.
Trim unlikely tokens Use one control, top_p at 0.9 to 0.95 or top_k at 40.
Stop the model repeating itself Set a repetition penalty of 1.0 to 1.1. Above 1.15 it breaks code.
Trap Temperature 0 is greedy, not reproducible. Batch size and kernel order can still change the output.
Reliable JSON and tool calls Force output to match a JSON Schema Constrained decoding, through response_format in vLLM or a GBNF grammar in llama.cpp.
Turn on tool calls in vLLM vllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes
Guarantee a tool call parses Set tool_choice to required or a named tool.
Trap A valid shape isn't correct content. Still validate the values, and retry on failure.
Open reasoning models Split the thinking from the answer vllm serve <model> --reasoning-parser deepseek_r1
Stop answers getting cut off mid-thought Raise max_tokens and the context limit. Thinking can run thousands of tokens.
Trap Follow the vendor's sampling settings. Greedy decoding tends to loop.
Serve many users batch requests on a GPU server Pick a runtime Serve one user on a laptop Ollama to start fast. llama.cpp to embed the engine or reach flags Ollama hides.
Serve many users on a GPU vLLM, with the widest model and hardware support.
Serve long shared prefixes or chained calls SGLang, whose RadixAttention caches shared prefixes.
Know where TGI stands In maintenance mode. Hugging Face now points to vLLM or SGLang.
Trap Don't judge by one request. llama.cpp looks fine alone, then time to first token climbs as users queue.
Stand up an OpenAI-compatible server Name the routes an OpenAI-compatible server copies /v1/chat/completions and /v1/models, with OpenAI's request and response shapes.
Serve under your app's model name, with a key vllm serve <model> --served-model-name gpt-4o-mini --api-key local-key
Point the app at it export OPENAI_BASE_URL=http://localhost:8000/v1 export OPENAI_API_KEY=local-key
Check it answers before changing app code curl -H "Authorization: Bearer local-key" localhost:8000/v1/models
Tip Compatible means the same shape, not every feature. Tool calls, structured output and logprobs vary by runtime.
Trap /v1/chat/completions needs a chat template. Pass --chat-template if the model ships none.
Continuous batching and PagedAttention static A idle C static B D continuous A C E continuous B D time Each letter is a request in one of two batch slots. Static batching leaves a slot idle until the longest request ends. Continuous batching refills it at once.
Explain continuous batching The batch changes every decode step. A finished request leaves and a waiting one joins.
Explain PagedAttention It stores the KV cache in 16-token blocks, so waste is at most one partial block.
Say what vLLM does when the cache fills It preempts a request and recomputes it later.
Trap KV-cache memory caps the batch, not compute.
Prefix caching Reuse the KV of a repeated prompt start vllm serve <model> --enable-prefix-caching On by default in vLLM V1.
Order the prompt so the cache hits Stable parts first, like the system prompt and documents. The user's question goes last.
Trap It matches from token 0. One changed character early invalidates everything after it.
Latency or throughput Define latency and throughput Latency is one request's wait. Throughput is tokens per second across all users.
Tune for latency vllm serve <model> --tensor-parallel-size 2 --max-num-seqs 16
Tune for throughput vllm serve <model> --max-num-seqs 256 --gpu-memory-utilization 0.95
Stop a long prompt stalling other requests vllm serve <model> --enable-chunked-prefill
Trap Frequent preemptions mean you pushed the batch too far. Each evicted request gets recomputed.
Speculative decoding Explain speculative decoding A small draft model guesses a few tokens. The big model checks them in one pass.
Know when it helps Batch size one, on predictable output like code or JSON.
Know when it backfires Big batches, where the GPU is already busy, and high-temperature prose.
Tip It's lossless. Verification keeps the output distribution the same, so only speed changes.
Fine-tunes & adapters train on your data, then serve it Fine-tune with LoRA or QLoRA Pick LoRA or QLoRA LoRA if the base fits on your GPU. QLoRA loads it in 4-bit if not.
Know what fine-tuning teaches Format, style or behavior. Use retrieval for new facts.
Pick starting settings The instruct model, hundreds to thousands of examples, rank 16, 1 to 3 epochs.
Trap Format the examples in the model's chat template, and stop when a held-out eval stops improving.
Adapters or merged weights Serve a base model with an adapter vllm serve <base> --enable-lora --lora-modules sql=/adapters/sql
Choose adapters Several fine-tunes on one GPU, or frequent new versions.
Choose merged weights One variant takes all traffic. Merge in fp16, then quantize.
Trap An adapter only works on its exact base checkpoint, so pin that version.
Hot-swap LoRA adapters Allow adapters to load at runtime VLLM_ALLOW_RUNTIME_LORA_UPDATING=True vllm serve <base> --enable-lora
Load an adapter without a restart POST its name and path to /v1/load_lora_adapter . Clients then send that name as model.
Cap adapters per batch vllm serve <base> --enable-lora --max-loras 4 --max-lora-rank 32
Trap Guard the load endpoint. It reads any path you hand it.
Scale across GPUs split a model across GPUs and plan capacity Split work across GPUs Split every layer across GPUs in one node vllm serve <model> --tensor-parallel-size 8
Add stages across nodes vllm serve <model> --tensor-parallel-size 8 --pipeline-parallel-size 2
Pick the order TP first, up to 8 GPUs on NVLink. Add PP only if it still doesn't fit.
Pick hardware for each phase Prefill wants FLOPs. Decode wants memory bandwidth and room for KV cache.
Run prefill and decode on separate GPU pools Disaggregated serving. It needs NVLink or InfiniBand, and pays off only at scale.
Trap TP size must divide the attention head count. PP needs enough requests to keep every stage busy.
Plan capacity and benchmark Little's Law in flight = 40 req/s x 6 s = 240 replicas = 240 / 40 per GPU = 6 + headroom
Find one GPU's capacity Raise the batch at real prompt lengths until p95 time per token breaks the budget.
Leave headroom Size for peak and keep replicas near 70% load. Queueing delay explodes above that.
Load-test a vLLM server vllm bench serve --model <model> --dataset-name random --request-rate 8
Report the right numbers TTFT, inter-token and end-to-end latency at p50, p95 and p99.
Trap Replayed prompts hit the prefix cache and inflate results. Turn it off unless production shares prefixes.
Evaluate, ship & monitor quality tests, rollouts and platform design Test a local model's quality Build a test set 50 to 200 real prompts with accepted answers.
Set the pass bar Before you see results. Grade the worst cases, not the average.
Pick what to measure Task quality, the share of valid JSON and tool calls, and p95 latency under load.
Trap A point or two of quality loss is fine. A 10% invalid-JSON rate is not.
Monitor a self-hosted model Scrape vLLM's Prometheus metrics curl -s localhost:8000/metrics
Spot saturation Watch queue wait and preemptions. GPU utilization stays high even as throughput drops.
Watch GPU health Use DCGM, NVIDIA's metrics agent, for power, throttling and ECC errors.
Trap Alert on latency percentiles per phase, never on averages.
Pin and roll out model versions Pin a model behind a stable name vllm serve <model> --revision <hash> --served-model-name assistant
List what one version pins Weights, tokenizer, chat template, quantization and engine version.
Roll out a new version Blue-green. Gate on the golden set, then shift a small share. Roll back with a router flip.
Trap Hub tags can move. A changed chat template alters output with no weight change.
Air-gapped deployment Stop runtime downloads and telemetry export HF_HUB_OFFLINE=1 export TRANSFORMERS_OFFLINE=1 export DO_NOT_TRACK=1
Bring the artifacts in Weights, images and wheels cross once, hash-verified and signed.
Trap Block egress at each node. A runtime that retries a download passes staging, then hangs inside the gap.
Many models, many regions Cut a multi-model GPU bill Cut the model list and share busy pools. Send bulk jobs to spot GPUs and rare models to a provider.
Route across regions By queue depth and cache locality. A queued GPU costs more than distance.
Trap A router without offline evals quietly sends everything to the biggest model.
Nothing matches. Try a shorter word, or clear the search.
Practice Guided practice One question at a time, with the ones you miss coming back. Practice 55 interview questions Full answers, from beginner to expert.
Found this useful? Pass it on.
Share on LinkedIn
Share on X
Copy link
This sheet stays free. With Pro , you can:
Every answer on this sheet. Read the full answer to all 55 questions it links, not just the 7 free ones.My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
