Fine-tuning takes a model that is already trained and keeps training it on a smaller, focused dataset. Pretraining from scratch starts from random weights and teaches the model language itself, from raw text at huge scale.
Pretraining from scratch
Fine-tuning
Starting weights
Random
An existing trained model
Data
Trillions of tokens of raw text
Thousands of curated examples
Compute
Weeks or months on large GPU clusters
Hours or days on a few GPUs
Goal
Build general ability and knowledge
Shape behavior, style, or format
The mechanism is the same in both: gradient descent on a next-token prediction loss. The difference is the starting point and the scale.
That starting point drives the practical rules. The base model already holds the ability you want, so fine-tuning uses a small learning rate and few passes. A high learning rate or too many passes erode the general ability pretraining built.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
When would you continue pretraining on raw domain text instead of running SFT?Use it when the base model barely saw the domain, like a new language or dense specialist text. Note that it needs far more tokens than SFT.
What you can say
Fine-tuning takes a model that's already trained and keeps training it on a smaller, focused dataset. Pretraining from scratch starts from random weights.
Pretraining is where the model learns language itself, from trillions of tokens of raw text, and that takes weeks or months on large GPU clusters.
Fine-tuning usually needs thousands of curated examples and hours or days on a few GPUs. Its goal is to shape behavior, style or format.
The mechanism is the same in both. It's gradient descent on a next-token prediction loss, and only the starting point and the scale differ.
The base model already has the ability you want, so you fine-tune with a small learning rate and only a few passes over the data.
If you push the learning rate too high or run too many passes, you erode the general ability that pretraining built.
Weak answers to avoid
Says fine-tuning uses a different training objectiveBoth run gradient descent on next-token loss. Say that what differs is the starting weights and the scale of data and compute.
Pitches fine-tuning as the way to add knowledgeA few thousand examples shape behavior, style and format. For new or changing facts, say you'd reach for RAG instead.
Skips why the learning rate stays smallThe base model already has the ability. A high rate or too many passes erodes it, so name catastrophic forgetting and the small-rate, few-pass rule.
Gives a one-line definition and no comparisonThe interviewer wants the contrast. Compare starting weights, data size, compute and goal so the gap between the two is concrete.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
