home
›
cheatsheets
›
ai_engineering
›
fine_tuning
Free to read Every card and diagram, and Quiz me
2
Free account Star cards. Your Quiz me marks are kept, so next time you drill just what you missed
3
Pro All 56 answers on this sheet, My sheet across every topic, and PDFs
A free account keeps your stars and Quiz me marks. Pro adds all 56 answers on this sheet, My sheet across every topic, and PDFs.
/
Read
Quiz me
Comfortable
Compact
0
Click any code to copy it
Done 0 /0
To review 0
On this sheet
01 When to fine-tune02 Supervised fine-tuning03 LoRA & QLoRA04 Training data05 Distillation06 Evaluate & debug07 Preference tuning & RL08 GPUs, tools & cost09 Tool calling & RAG10 Serve & roll back
The model Fine-tuning takes a trained model and keeps training it on your examples, with pretraining's next-token loss and a small learning rate.Fine-tuning changes how a model answers, like its format, tone and tool calls. Use RAG for what it should know. LoRA freezes the base model and trains a small adapter beside it. QLoRA stores that base in 4-bit, so an 8B model fits a 24 GB GPU.Data quality matters more than volume. A few hundred clean rows from real traffic, all in one format, beat thousands of scraped ones. Training loss always falls. Judge a fine-tune on held-out evals against the prompted base model, and check what it forgot. A checkpoint ships only if it beats the prompted base model on held-out evals. Real traffic feeds the next round.
When to fine-tune prompting, RAG, fine-tuning or pretraining Fine-tuning vs pretraining Say what fine-tuning starts from A trained model. Pretraining starts from random weights. Both use next-token loss.
Tell instruction tuning from task tuning Breadth. Instruction tuning spans many tasks. Task tuning trains one narrow job.
Trap A high learning rate or too many epochs erodes what pretraining built.
Good at behavior, bad at facts Name what fine-tuning is good at Format, tone, one narrow task, tool-call shapes, and moving work to a smaller model.
Say why it's bad at new facts A small run learns the shape of an answer, then invents the details.
Update a fact that changed RAG needs one document edit. A fine-tune needs a new training run and eval.
Trap Training on facts the base never saw rewards confident guessing. Hallucination can rise elsewhere too.
Prompt, RAG, fine-tune or pretrain Pick the first thing to try Prompting. Stay there while you can put the behavior into words.
Give the model private or fresh facts RAG. It changes the context, not the weights.
Know when to fine-tune When prompts work but are long, slow or unreliable.
Know when to run continued pretraining For text the base model barely understands. It takes billions of raw tokens, then SFT again.
Tip These stack. Most systems run prompting plus RAG, and add a fine-tune later for cost or consistency.
Pick a base model Pick the model size The smallest one that already does the task sometimes, with a good prompt.
Choose the instruct or the base checkpoint Instruct for a small style shift. Base for a big dataset or a custom format.
Choose between two or three candidates Tune each on the same 1,000 examples and compare them on your own eval.
Tip Check the license, context length, tokenizer coverage and serving support first.
Trap Fine-tuning rarely creates a skill the base model lacks. It sharpens one that's already there.
Supervised fine-tuning what one step does, and the settings to start with One SFT step Say what the loss measures Cross-entropy. It measures how surprised the model was by each of your actual tokens.
Update the weights out.loss.backward() optimizer.step() Backprop gives each weight a gradient, and the optimizer takes a small step.
Trap Fine-tuning stores nothing and looks nothing up. It makes answers like yours more likely.
Mask the loss Score only the assistant tokens labels[:len(prompt_ids)] = -100 Cross-entropy skips any label set to -100.
Say why masking matters A 2,000-token prompt would drown the signal from a 50-token answer.
Teach the model to stop Keep the end-of-turn token in the loss. Mask it and the model never learns to stop.
Tip Mask padding too. Then decode one batch and check which positions are -100.
Match the chat template See the exact string the model trains on tok.apply_chat_template(messages, tokenize=False)
Catch a doubled BOS token The trainer and the template both add one. It's invisible in the JSONL and costs accuracy.
Check that training matches serving Render a row with the serving tokenizer and diff it against a real request.
Trap A template mismatch between training and serving fails silently. The outputs just get worse.
Starting hyperparameters Set the learning rate learning_rate: 2.0e-4 1e-4 to 2e-4 for LoRA. 1e-5 to 2e-5 for full fine-tuning.
Pick the number of epochs num_train_epochs: 3 2 to 3. Small datasets memorize fast after that.
Get an effective batch of 32 per_device_train_batch_size: 4 gradient_accumulation_steps: 8 Aim for 16 to 64 sequences.
Set the max sequence length Cover your 95th-percentile row, or truncation cuts off answers.
Tip Cosine decay with 3 to 5% warmup rarely needs tuning. Change one setting at a time.
LoRA & QLoRA train a small adapter, freeze the rest How LoRA works x W frozen A B + h LoRA layer x runs through frozen W and through A then B. The sum is output h. Only A and B train.Write the LoRA forward pass h = x @ W.T + (alpha / r) * (x @ A.T) @ B.T W stays frozen. The rank r is often 8 to 64.
Say why training starts at the base model B starts at zero, so the update BA is zero on step one.
Say why LoRA became the default One mid-range GPU, megabyte adapters, and near full quality on narrow tasks.
Set rank, alpha and targets Start with a standard config LoraConfig(r=16, lora_alpha=32, target_modules="all-linear")
Say what rank and alpha control Rank sets capacity. Alpha only scales the update, by alpha / r.
Add capacity for a new skill Raise rank to 32 or 64 and keep alpha at 2r, so the step size stays the same.
Choose which layers to adapt Every linear layer, in attention and the MLP. The MLP holds most of the weights.
Tip Coverage beats rank. In QLoRA's tests, LoRA matched full fine-tuning only on all linear layers.
Trap Leave the embeddings and lm_head out unless you add new tokens. Those matrices are huge.
QLoRA Say what QLoRA changes The frozen base is stored in 4-bit NF4. The adapters stay in bf16.
Name QLoRA's three innovations NF4 for normally distributed weights, double quantization, and paged optimizers.
Choose QLoRA over LoRA When the model won't fit otherwise. Dequantizing makes every step slower.
Trap Don't merge the adapter into 4-bit weights. Merge it into the 16-bit base, or serve it separately.
Full fine-tuning and other PEFT methods Know when full fine-tuning is worth it A new language or domain, a very large dataset, a small model, or one model to ship.
Name PEFT methods besides LoRA Bottleneck adapters, prompt tuning, IA3, BitFit and DoRA.
Get closer to full quality at low rank DoRA. It learns a magnitude per weight and uses LoRA for the direction.
Trap Bottleneck adapter layers never merge into the weights, so they add latency forever.
Training data rows, size, mix, splits and cleaning Write the training rows {"messages": [ {"role": "system", "content": "Reply in JSON."}, {"role": "user", "content": "Charged twice."}, {"role": "assistant", "content": "{\"type\": \"billing\"}"} ]} one line in the real file
Keep every row consistent Same system prompt and output shape on every line. The model copies sloppiness too.
Pick where the inputs come from Real traffic, so training inputs match production.
Tip Ship two files, one for training and one for validation.
How much data, in what mix Size the dataset 50 to 500 rows for tone or format, 500 to 5,000 for a narrow task, 10,000 or more for a domain.
Find out if more data would help Train on 25%, 50% and 100% and plot the eval score. Collect more only if it still climbs.
Mix a domain fine-tune About 70% domain, 20% general replay, 5% safety, 5% hard cases, weighted by tokens.
Split and deduplicate Hold out a validation split 5 to 10% after dedupe, split by document, customer or template.
Remove near-duplicates MinHashLSH(threshold=0.8, num_perm=128) Run it after an exact pass that hashes normalized text.
Dedupe on the prompt too Two answers to one prompt conflict, and the model learns to hedge.
Keep eval items out of training Match 13-word n-grams and embedding neighbors. Keep a private holdout nothing touches.
Trap Duplicates act as extra epochs. The model memorizes them and repeats them word for word.
Distillation a bigger model writes the training data Distill a big model into a small one Say what distillation is A large teacher writes answers for your task, and you fine-tune a small student on them.
Tell hard labels from soft labels Hard labels are the teacher's text. Soft labels match its token probabilities and need its logits.
Stop the student drifting at inference On-policy distillation. The student samples, and the teacher scores those samples.
Replace a frontier model with a small one Only for a narrow, stable task. Shadow it, canary 1 to 5%, and route rare inputs back.
Trap The student copies the teacher's mistakes. Judge it on real ground truth, and check the license.
Generate synthetic data Pick the safest way to generate rows A stronger model answers real production prompts. Invented prompts drift.
Filter what the model wrote Keep rows that pass a verifier, like a unit test, a schema check or human review.
Avoid model collapse at scale Keep real data in every mix, and a human-written holdout you never generate from.
Trap Never verify with the model that wrote the data. Its own mistakes pass its own check.
Evaluate & debug prove it's better and find what broke Prove the fine-tune is better Pick the baseline What you run today, usually the prompted base model, on held-out examples.
Score the outputs Task metrics, blind pairwise wins for open-ended answers, and a regression set.
Size the comparison Hundreds of examples, with a confidence interval. A 3-point gain on 50 examples is noise.
Test format adherence One validator per constraint, each with its own pass rate. Score the worst of several samples.
Trap Offline wins that don't move real user metrics aren't wins. Ship behind a shadow or A/B test.
Spot overfitting Read the loss curves Training loss falls while validation loss rises. Stop at the best checkpoint.
Fix outputs that quote training rows Dedupe, then add more varied prompts.
Fix answers that collapse to one template Lower the learning rate and the LoRA rank or alpha, and add dropout.
Tip If held-out evals drop but validation loss looks fine, suspect leakage across the split.
Trap Rising validation loss alone doesn't prove a worse model. Gate on task evals.
Forgetting and lost safety Define catastrophic forgetting A narrow fine-tune degrades skills the model had, because the loss rewards only your data.
Prevent forgetting Replay 5 to 30% general data. Use low-rank LoRA, a low learning rate and 1 to 3 epochs.
Say why fine-tuning can break safety Alignment is shallow. Ten harmful examples jailbroke GPT-3.5 Turbo.
Keep refusals working Mix in 2 to 5% refusals, keep the template unchanged, and gate on a harmful-prompt set.
Tip Keep a retention eval beside the task eval, and gate releases on both.
Trap Benign data weakens safety too, and LoRA doesn't protect it.
Debug a model that got worse Check which tokens get trained tok.decode(labels[labels != -100]) It should show assistant text plus EOS.
Fix a model that never stops EOS is missing from the targets, or masked out of the loss.
Fix a model that ignores instructions The training and serving templates differ. Compare them byte for byte.
Fix a model that's fluent but dumber The learning rate was too high, or the run had too many epochs.
Tip Check formatting before hyperparameters. Endless rambling almost always means broken formatting.
Trap A misnamed adapter fails silently, and you serve the base model. Check that it loaded.
Preference tuning & RL teach the model which answer is better RLHF and DPO Name the three stages of RLHF SFT on demos, a reward model trained on ranked pairs, then RL against it with a KL penalty.
Name RLHF's costs Slow labels, four models in memory, an unstable loop, reward hacking and an alignment tax.
Say what DPO removes The reward model and the RL loop. It trains straight on chosen and rejected pairs.
Write the DPO loss -logsigmoid(beta * (chosen_logratio - rejected_logratio)) Each log-ratio is the policy's log-prob minus the frozen reference's.
Tip DPO runs like SFT, with two models in memory, not four. It learns only from the pairs you collected.
Build preference pairs Pick where the answers come from Sample both from the model you're about to train, on real production prompts.
Stop DPO learning that longer wins Match length and format across the pair, and track the length ratio.
Keep the judge honest Use a different model family, recalibrated on human labels each round.
Tip Release on a held-out human comparison, never on training loss or judge score alone.
Trap Pairs from a much stronger model teach a gap your policy can't close.
DPO's successors and GRPO Name DPO's failure modes The chosen answer's likelihood drops too, longer answers win, and the policy drifts.
Match IPO, KTO, ORPO and SimPO to their fix IPO curbs overfitting, KTO needs no pairs, ORPO folds in SFT, and SimPO fixes length bias.
Say what GRPO drops from PPO The critic. The mean reward of a group of sampled answers becomes the baseline.
Reward answers a program can check RLVR. Unit tests, exact match or schema checks replace a learned reward model.
Tip Run SFT first, then GRPO with verifiable rewards for math and code. Keep DPO for open-ended taste.
Serve & roll back deploy adapters, version them, keep improving Merge the adapter or serve it Merge an adapter into the base merged = model.merge_and_unload() For one variant, at base speed.
Serve many variants on shared GPUs Keep adapters separate, 10 to 200 MB each. A new version is a pointer change.
Batch requests for different adapters The base matmul runs once. A grouped kernel like SGMV applies each request's adapter.
Tip Keep adapters in CPU RAM and page them in with an LRU cache. A cold adapter slows its first request.
Trap Every adapter must share the same base model. A merged variant is a full copy, one per GPU.
Version, release and roll back Version a fine-tune Keep weights, data hash, template hash, config and eval report as one immutable version.
Roll back in seconds Serve through an alias like prod. Keep the previous version loaded and flip the pointer.
Turn production traffic into training data Log with consent, scrub personal data, and curate from ratings, edits and abandons.
Release a retrained model Pass a fixed and a fresh eval suite, then go shadow, a 5% canary, and full traffic.
Trap Roll back the prompt template with the weights. A new prompt on old weights is an untested pair.
Nothing matches. Try a shorter word, or clear the search.
Practice Guided practice One question at a time, with the ones you miss coming back. Practice 56 interview questions Full answers, from beginner to expert.
Found this useful? Pass it on.
Share on LinkedIn
Share on X
Copy link
This sheet stays free. With Pro , you can:
Every answer on this sheet. Read the full answer to all 56 questions it links, not just the 7 free ones.My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
