Deep Learning & Neural Networks.
Cheat sheet.

• Forward pass computes the loss; backprop multiplies local derivatives layer by layer (chain rule); the optimizer steps weights against the gradient.
• Depth only buys you something if a non-linearity sits between layers: W2(W1x) = (W2W1)x. If a deep model scores exactly like logistic regression, the activations are missing.
• Almost every knob you turn is one of three things: step size (lr, schedule), gradient noise (batch size), or capacity/constraint (width, depth, regularization).
• Training memory = params + grads + optimizer state + stored activations. That last term is why a batch that fits at inference OOMs during training.

Match head, activation and loss to the task

Never apply softmax/sigmoid before CrossEntropyLoss/BCEWithLogitsLoss - they take logits. Double-squashing trains, but badly, and silently.

Sanity-check loss at step 0

Far off at step 0 means wrong label encoding or wrong head shape, not a bad model.

Fix class imbalance

CrossEntropyLoss(weight=w) or BCEWithLogitsLoss(pos_weight=n_neg/n_pos). Also init the final bias to log(p/(1-p)) so the model starts at the base rate instead of spending epochs getting there.

Many dead ReLUs (a whole channel outputting 0 for all inputs) usually means the learning rate is too high, not that ReLU is wrong.

Start from these defaults

Adam converges faster; well-tuned SGD+momentum often generalizes slightly better.

Tune in this order

learning rate -> schedule/warmup -> batch size -> regularization strength -> depth/width. Move lr by factors of 3 or 10, not 10%. Sweeping everything at once wastes compute, because lr swamps the rest.

Pick a schedule

• Warmup for the first 1-5% of steps (or 500-2000 steps), linear from 0.
• Then cosine decay to ~0, or step decay (x0.1 at 50%/75%).
• Linear scaling rule: multiply batch size by k -> multiply lr by k, and add warmup or the first steps explode.

Find a starting lr in one run

LR range test: ramp lr exponentially over ~100 steps, plot loss, take roughly one order of magnitude below the lr where loss starts diverging.

Get the update order right

model.train()
for xb, yb in train_loader:                      # shuffle=True
    optimizer.zero_grad(set_to_none=True)
    loss = criterion(model(xb), yb)              # criterion takes logits
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()                             # per-step schedules only

Choose a batch size

• Start with the largest batch that fits comfortably, then confirm validation still holds up.
• Typical range 32-512. Below ~16 per device, BatchNorm statistics go unreliable - switch to GroupNorm or LayerNorm.
• Small batches inject noise that acts as mild regularization; very large batches need a larger lr plus warmup and can land in sharp minima that score worse held-out.

Estimate training memory

Activations scale with batch size x sequence/resolution and dominate for big inputs.

Cut memory without shrinking the model

Work through the fixes in this order

1. Verify the split is honest - leakage, duplicate rows, time-shifted validation all fake this curve.
2. Early stopping with best-checkpoint restore (patience 5-10). Free, recovers most of the gap.
3. More data or augmentation. Beats every regularizer when available.
4. Then raise weight decay, add dropout to the head, or shrink the model.

Pick the right regularizer

Dropout stacked on BatchNorm often hurts - they disagree about variance. Pick one per block.

Keep dropout out of your validation numbers

Dropout is off at inference and activations are rescaled automatically. Forgetting model.eval() leaves it active, so validation looks noisy and worse than it is.

Pick the normalization axis

Stop BatchNorm from breaking at serving time

Training uses the current batch's mean/variance; inference uses running averages. Call model.eval() (or training=False). Symptoms of getting it wrong: the same input gives different answers depending on batch composition, and single-example inference is garbage.

Freezing a pretrained backbone with requires_grad=False does not freeze its BN statistics. Put the backbone in .eval() too, or running stats drift while the frozen weights still expect the old ones.

Know why transformers use LayerNorm

Variable sequence lengths and padding make batch statistics meaningless, autoregressive decoding runs with an effective batch of 1, and layer norm needs no cross-device sync. Pre-LN blocks (norm before attention) train stably; post-LN needs careful warmup.

Scale matters more than distribution: too small and activations collapse with depth, too large and they saturate. Normalization plus residuals make this forgiving on modern architectures; it still bites deep nets without norm and RNNs.

Clip by global norm across all parameters, not per element - norm clipping keeps the update direction and only shortens the step. With AMP, scaler.unscale_(optimizer) before clipping.

Compute the output size

out = (n + 2*p - k) // s + 1
# n=32, k=3, p=1, s=1 -> 32   (size preserved)
# n=32, k=3, p=1, s=2 -> 16   (halved)
# dilation d: replace k with d*(k-1) + 1

"Same" padding at stride 1: p = (k-1)//2 -> 3x3 uses p=1, 5x5 p=2, 7x7 p=3.

padding='same' does not preserve size once stride > 1, and even-sized kernels cannot pad symmetrically - the framework pads one side more and shifts your features.

Count conv parameters

(k * k * C_in + 1) * C_out. A 3x3 conv, 64 -> 128 channels: (9*64 + 1) * 128 = 73,856. Compare a fully connected layer on a 200x200 image into 1000 units: 40 million weights.

Reach for the right layer

Max pool asks "did this feature fire nearby"; average pool keeps the general level. Max inside the network, global average at the end. Pooling has no learned weights.

Know why plain RNNs fail on long inputs

The same weight matrix multiplies the gradient at every timestep, so 100 steps means 100 similar factors multiplied. Below 1 the signal dies, above 1 it explodes. Dependencies more than a few dozen steps back never get learned.

Read the LSTM gates

The cell state updates by addition, which is the whole reason gradients travel further than in a plain RNN. It stretches the useful range; it does not remove the limit.

Choose LSTM vs GRU

GRU = 2 gates, no separate cell state, ~25% fewer parameters, slightly faster. Pick GRU for limited data, short sequences, tight compute. Pick LSTM for long sequences needing precise memory. The gap is usually smaller than what tuning the learning rate buys.

Batch variable-length sequences

Pad, then pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False) so the RNN skips pad steps and the final state is the real last step. Mask padded positions out of the loss too.

Recall the attention formula

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

Query and key dims must match (dot product); the value dim is free.

Remember why the scores are scaled

Dot products of d_k-dim vectors have variance d_k. Unscaled, one score dominates, softmax saturates into near one-hot, and its gradient collapses. In fp16 the logits can overflow before softmax even runs.

Split the model dimension across heads

d_model = n_heads * d_head -> 8 heads x 64 = 512, costing about what one 512-dim head costs. More heads = more relations tracked, each in a smaller subspace. Head count is tuned, not maximized.

Mask the future in a decoder

Add a strictly-upper-triangular matrix of -inf to the scores before softmax, so illegal weights become 0. Without it, position 5 attends to position 6 and copies the answer: training loss looks wonderful and generation is nonsense.

Give the model a sense of position

Attention is permutation equivariant - with no position signal, "dog bites man" and "man bites dog" encode identically.

Budget attention cost

• Time and memory: O(n^2 * d) in sequence length. Doubling context quadruples attention work.
• KV cache bytes ≈ 2 * n_layers * n_kv_heads * d_head * seq_len * batch * bytes_per_elem. This is where most inference memory goes; grouped-query attention shrinks n_kv_heads specifically to cut it.

Pick the architecture family

Tokenize correctly

• Always serve a model with the exact tokenizer it was trained with. Different ids mean the input is nonsense.
• BPE merges the most frequent adjacent pair repeatedly until the vocab hits its target size. Frequent words stay whole; rare ones split.
• Rough budgeting for English: 1 token ≈ 4 characters ≈ 0.75 words.
• Merges follow frequency, not morphology - code identifiers, numbers, and languages thin in the training corpus fragment badly and cost extra tokens.

Read an embedding space

• Similarity is cosine (angle), not Euclidean distance.
• Closeness means "appears in similar contexts", which is not meaning: hot and cold sit close because they fill the same slots. Sentiment work built on cosine similarity breaks on this.
• word2vec/GloVe give one static vector per word, so senses of "bank" are averaged and unseen words have no vector at all. Use contextual embeddings from a transformer when polysemy matters.

Follow the standard recipe

1. Replace the output head to match your class count.
2. Freeze the backbone (and put it in .eval() so BN stats stay put), train the head at ~1e-3.
3. Unfreeze and fine-tune everything at 10-100x lower lr (1e-5 to 1e-4), watching validation.
4. Optional: discriminative lrs, lower for earlier layers.

Know when it pays off

Pays off with scarce labels and a source domain close to yours (photos -> photos). Pays off less across a domain gap (photos -> medical scans). With millions of task-specific examples, from scratch can match it. Failure mode is negative transfer: aggressive fine-tuning on tiny data overwrites the representations you wanted.

Run these checks before touching the architecture

Read the two loss-curve failure modes

• Missing optimizer.zero_grad() - gradients accumulate silently across steps.
• Missing model.eval() and torch.no_grad() at validation - dropout and BN corrupt the numbers.
• Softmax applied before CrossEntropyLoss, or one-hot float targets where int class ids are required.
• Normalization statistics computed over train + validation together (leakage).
• shuffle=True missing on the train loader.
• Batch size changed without rescaling the learning rate.
• Frozen backbone whose BatchNorm running stats are still updating.
• Tokenizer or image preprocessing at serving that differs from training.
• Gradient descent has no guarantee of a global optimum on a non-convex surface, and in high dimensions most stationary points are saddles, not bad minima. The target is a flat low region that generalizes, not the lowest point - removing gradient noise can improve train loss and hurt test loss.