Supervised fine-tuning is the same next-token prediction as pretraining, run on your examples. The weights already exist, so each step only nudges them.
Each row is rendered with the chat template, then tokenized. The model does one forward pass and predicts a distribution over the next token at every position. Cross-entropy loss measures how surprised it was by your actual tokens. Backpropagation then works out how much each weight contributed to the loss. That gives a gradient, and the optimizer subtracts a small step.
for batch in loader: # one optimizer step per batch
out = model(input_ids=batch["input_ids"], labels=batch["labels"])
out.loss.backward() # gradient for every trained weight
optimizer.step() # weights move a little
optimizer.zero_grad()
Nothing is inserted or looked up. The model becomes more likely to produce sequences that look like your answers. Fine-tuning learning rates are much smaller than pretraining ones, so each pass shifts behavior gently.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓