Classical ML Algorithms.
Cheat sheet.

Check whether this model needs feature scaling

Fit the scaler on the training fold only. Fitting on train+test leaks distribution info.

Read a linear coefficient correctly

Expected change in the target per +1 unit of that feature, others held fixed. Intercept = prediction when every feature is 0, often far outside the data and meaningless.

• Raw coefficients are not comparable across features (millimetres earn tiny numbers). Standardize first to rank influence.
• "Others held fixed" is fiction when features move together, so it is not a causal effect. Signs can flip when you add or drop a correlated feature.
• Always report a coefficient with the exact feature set it was fitted on.

Diagnose broken regression assumptions from one plot

Residuals vs fitted values:

Linearity and independence corrupt the estimates. Constant variance and normality only corrupt the uncertainty around them.

Turn a logistic coefficient into an odds ratio

import math
math.exp(0.7)    # 2.01 -> odds roughly double per +1 unit

• log-odds = ln(p / (1-p)); logistic regression is linear in that space.
• A coefficient is not a probability change. Odds 0.01 -> 0.02 barely moves p; odds 1 -> 2 moves p from 0.50 to 0.67.
• Report odds ratios, and quote a probability only at a stated baseline.

Remember what the sigmoid does to gradients

1 / (1 + math.exp(-2.2))   # 0.900   (z=0 -> 0.5, z=-2.2 -> 0.100)

Saturates: once |z| is large the curve is flat and the gradient nearly vanishes, so a confidently wrong prediction learns very slowly. Unscaled features push scores into that flat region and training stalls.

Decide between the normal equation and gradient descent

• Closed form: one step, roughly O(p^3), needs X'X invertible. Perfectly correlated columns make it singular, near-collinear ones make it unstable. Good libraries use QR or SVD, never a literal inverse.
• Gradient descent when p is large, data does not fit in memory, updates stream, or the loss has no closed form (logistic, lasso, anything boosted).

Detect and handle multicollinearity

Symptom: coefficients swing wildly across resamples and signs flip against intuition, long before accuracy degrades.

Pick L1 vs L2 vs Elastic Net

Lasso keeps roughly one member of a correlated group at random, and can never select more features than you have rows. The L2 part of Elastic Net fixes both.

Choose the regularization strength

• Log-spaced grid (1e-3 to 1e2, times 10 or times 3 per step), cross-validated. Linear grids waste most of their points.
• Pick by the metric you care about, never training loss - that always prefers the weakest penalty.
• One-standard-error rule: take the strongest lambda scoring within 1 SE of the best. Simpler model, almost no accuracy cost.
• The CV score at the winning lambda is optimistic. Keep a separate test split for the number you report.

Recall how a split is chosen and what "pure" means

Greedy scan of every feature x every candidate threshold. Score = parent impurity minus the size-weighted average child impurity. Impurity is Gini or entropy (classification), variance (regression). Pure = one class only, impurity 0. Optimal trees are NP-hard, so greedy is the accepted compromise.

Stopping only at purity is memorization - the tree can always isolate single rows. Cap max_depth, set min_samples_leaf, or prune.

Gini favours high-cardinality features (more thresholds means more chances to look good). Watch identifier-like columns.

Know what each random forest randomization buys

• Bagging: each tree gets its own bootstrap sample (n draws with replacement), so structures diverge.
• Feature randomness: each split sees only a random subset, sqrt(p) for classification, p/3 for regression. Without it one dominant feature roots every tree, errors correlate, and averaging barely helps.
• Trade-off: a smaller subset decorrelates better but weakens each tree. Worth tuning.

Use out-of-bag error instead of a holdout

Each row is scored only by the trees whose bootstrap sample excluded it. Behaves like k-fold CV for the price of one training run.

(1 - 1/10_000) ** 10_000   # 0.3679 -> 1/e, so ~37% of rows are OOB per tree

• Reads slightly pessimistic on small forests (each row judged by ~1/3 of the trees).
• Assumes independent rows, so it silently leaks on time series or repeated entities. Use a grouped split there.

Recall the gradient boosting loop

Start at a constant. Each round: compute the loss gradient per row, fit a shallow tree to those pseudo-residuals, add it to the running prediction scaled by the learning rate.

• Squared error: the gradient is exactly the residual, so "fit the residuals" is literal.
• Log loss: it is a pseudo-residual, and leaf values are chosen to reduce the actual loss.
• Nothing in the procedure knows a residual is measurement error, so it chases label noise forever. Early stopping on a held-out set is not optional.

Tune a booster in the right order

1. Fix learning_rate around 0.05, let early stopping choose n_estimators. Never tune the two independently - halving the rate roughly doubles the trees needed for a small gain.
2. max_depth 3-8 (this caps interaction order) or num_leaves.
3. Row and column subsampling 0.5-1.0 - cuts variance and speeds every round.
4. min_child_weight / min samples per leaf, plus L2 on leaf weights, to stop tiny leaves memorizing rare rows.

Do not tune on the same split you early-stop on. Use a third holdout, or nested folds when data is small.

Choose forest vs boosted trees

Boosting usually wins by a few points on tabular data, not by a category. If a forest ties a tuned booster, ship the forest.

Recall what XGBoost changed

• Second-order objective (gradient + Hessian); each leaf value is a Newton step. Leaf L2 penalty plus a fixed cost per split, so pruning falls out of the objective.
• Speed is data layout: features pre-binned into histograms, compressed column blocks, one linear scan per level, threaded, cache friendly, spills to disk.
• Missing values get a learned default direction - on wide sparse matrices that alone is most of the win.
• Rounds are still sequential. More cores never shorten a 1000-round chain.

Recall how KNN predicts and what it cannot do

Store the rows, predict from the k nearest. Classification = majority vote; regression = mean of the neighbours' targets, optionally weighted by 1/distance. No training step; roughly O(n*d) distance computations per query.

• Prediction surface is a staircase, not a curve.
• Cannot extrapolate: past the training range it returns the same edge average, flat forever. Bad for trending data like prices over time.

Choose k

• Sweep k on a validation set, plot error, pick the flat region. sqrt(n) is a starting point, nothing more.
• Small k tracks noise (one mislabelled neighbour flips the answer). Large k smooths across the boundary and drags toward the majority class.
• Odd k only breaks ties in binary problems - with 3 classes a 2-2-1 vote still ties at k=5.
• Distance weighting matters more than parity: it makes ties rare and stops large k washing out local structure.
• On imbalanced data check per-class recall; raising k quietly buries the minority.

Choose a distance metric

The metric is a real hyperparameter - change it and the neighbour set, and so the prediction, changes.

Scale features before KNN

Distance sums squared differences, so the widest-range feature owns every decision. A 20-year age gap versus a 5000 salary gap is 400 against 25,000,000 - the model is a salary lookup wearing a KNN costume.

Do not standardize by reflex: it gives pure noise columns equal say. Min-max for bounded features, robust scaler for heavy tails.

Know why KNN dies at scale

The cost sits at serve time, and the model is the data, so memory grows with the dataset you keep collecting. KD-trees and ball trees get you near O(log n) in low dimensions but degrade to a full scan past roughly 20 features. Escapes: subsample, cluster the training set into representatives, or move to a model that compresses training into weights.

Know why high dimensions break neighbours

Distances concentrate - the ratio of farthest to nearest drifts toward 1, so the top neighbour is barely closer than the hundredth and the vote is near random. Irrelevant features add distance and no signal, so adding features often makes KNN worse. Fix order: select features, weight dimensions by relevance, PCA, or change model. Cosine helps on sparse text.

Recall the margin and support vectors

Hyperplane: w . x + b = 0. Margin is the gap to the closest point on each side. Maximizing it is convex, so the boundary is unique and reproducible run to run (perceptron-style rules return whatever they stumble into). Support vectors are the points on the margin edge - they alone determine the boundary; delete every other row, refit, get the same model.

Margin is measured in feature units, so it means nothing without scaling. A handful of borderline points control everything - one mislabelled edge row visibly shifts the boundary.

Set C (soft margin and slack)

Hard margin demands zero training errors and only exists on perfectly separable data; one bad label makes it infeasible. Soft margin allows slack per point: 0 = outside the margin and correct, between 0 and 1 = inside the margin but correct, above 1 = misclassified.

Always tune C. Do not assume hard margin generalizes even on separable data.

Use the kernel trick, and know when not to

A kernel returns the dot product two points would have in a higher-dimensional space without building that space. RBF corresponds to infinite dimensions in one exponential, which buys curved boundaries from a linear algorithm.

• gamma too high memorizes individual points.
• Training scales O(n^2) to O(n^3) and prediction touches every support vector, so it stalls past roughly 100k rows. On large tabular data a linear model or boosted trees win on both score and time.

Know what the independence assumption costs

Features are assumed independent given the class, so per-feature probabilities multiply instead of estimating a joint distribution. Almost always false ("new" and "york"), but the class ranking survives a lot of wrongness, so classification stays usable while the probabilities do not. Correlated features act as repeated votes, saturating scores near 0 or 1. Treat the output as a rank, never as calibrated confidence you can threshold on.

Set Laplace smoothing (alpha)

Any feature-class pair never seen together gets probability 0, and one zero factor drags the whole posterior to 0 - a single unknown token vetoes an obvious spam.

P = (count + alpha) / (total + alpha * V), V = vocabulary size.

alpha=1 is convention, not law. On a 100k-token vocabulary, adding 1 to every cell floods rare classes with fake evidence and flattens the model. Sweep it; small values often beat 1 on large vocabularies.

Recall what cross-entropy penalizes

Negative log of the probability assigned to the true class.

-math.log(0.9)   # 0.105  barely punished
-math.log(0.1)   # 2.303  over 20x worse

Averaged over the dataset it equals the negative log-likelihood, so minimizing it is maximum likelihood estimation. It cares about calibrated confidence, not just which class wins.

Know why not MSE for classification

Paired with a sigmoid, MSE gives a non-convex surface with flat regions: confidently wrong means the sigmoid derivative is near 0, the gradient vanishes, and learning stalls. Cross-entropy cancels that derivative - the gradient reduces to (prediction - label), so confidently wrong produces a large gradient. MSE is still fine as an evaluation score for probabilities (the Brier score); it is just the wrong training loss.

Diagnose what imbalance actually did

The loss sums over rows, so the majority class supplies most of the gradient. Boundaries drift into the minority region, and predicted probabilities pull toward the base rate, so a 0.5 threshold may never be crossed.

Imbalance only bites where classes overlap. With clean separation, a 1-in-1000 split trains fine. Check first: is ranking quality (PR-AUC) bad, or is only the threshold wrong?

Choose an imbalance fix

Start with class weights + threshold moving. Reach for SMOTE only after those plateau, and only inside the training fold.

Build a classifier when positives are under 1%

• Count positives in absolute terms first. 1% of 10M = 100k positives, which is plenty. 1% of 3k = 30, and no algorithm rescues that - aggregate features, use rules or anomaly detection, and buy labels.
• Default with enough positives: boosted trees with class weights, keeping every majority row that fits in memory.
• If you must undersample, recalibrate the predicted probabilities or cost-based decisions become meaningless.
• Rank by score and cut where a false positive costs what a catch is worth.
• Monitor the positive rate and score distribution. Rare classes drift, and a stale threshold quietly sends the wrong volume to review.

Cross-validate imbalanced data without leaking

• Resample inside each training fold, never before the split. Wrap sampler + estimator in a pipeline so it refits per fold.
• Oversampling first puts duplicate rows on both sides; a SMOTE point built from a held-out neighbour carries that neighbour's position into training.
• Score on the untouched, still-imbalanced fold - only that resembles production.
• Stratify the splits or a fold lands with zero positives and an undefined metric. Few positives -> repeated stratified splits, since one 5-fold run has huge variance.
• Repeated customers or events need a grouped split too.

Choose One-vs-Rest or One-vs-One

Recall the K-Means loop

Assign every row to the nearest centroid (squared Euclidean), then move each centroid to the mean of its assigned rows. Repeat until nothing moves. Each pass is O(nkd), so it stays fast on millions of rows. The mean is exactly the point minimizing within-cluster squared distance, so the objective only falls - to a local optimum.

Fix bad initialization

Two seeds landing inside one true cluster split it and merge two others. The loss goes flat, so it looks converged and stable. k-means++ picks seeds one at a time with probability proportional to squared distance from the nearest chosen centroid, at one extra pass per centroid. It reduces bad seeds, it does not remove them: run 10 restarts and keep the lowest inertia. If those restarts disagree badly, the problem is k or cluster shape, not seeding.

Choose the number of clusters

• The elbow is weak. Within-cluster sum of squares always falls, so you are hunting a bend, not a minimum, and on real data three people read three elbows.
• Silhouette (own cluster vs nearest other cluster) has a real maximum. The gap statistic compares your curve against uniform random data. Both still assume your algorithm's cluster shape.
• Stability check: rerun on subsamples. A k whose groupings change is not real.
• Cap k by the downstream use - segments a team can actually run campaigns for.

Recognize when K-Means is the wrong tool

It assumes round blobs of similar size, and every boundary is a straight line halfway between two centroids. Rings, crescents and elongated bands get sliced across their true shape while the loss still looks fine. The mean is not robust either: one far point drags its centroid off the dense region, and that shifted centroid then steals nearby points.

If the geometry really is spherical and clean, K-Means remains the fastest sensible choice - do not abandon it by reflex.

Choose K-Means or hierarchical

K-Means: needs k up front, O(nki*d), scales to millions, seed-dependent. Agglomerative: no k up front, roughly O(n^2 log n) time and O(n^2) memory, deterministic, and produces a dendrogram you can cut at any height - which matters when the domain is genuinely nested (taxonomies, document topics). Past tens of thousands of rows it stops being practical.

Middle path: cluster a sample hierarchically to read off k, then run K-Means on everything.

Pick a linkage rule

Linkage changes the result more than the distance metric does. Compare single against Ward when you suspect elongated structure.

Set DBSCAN's eps and minPts

A point is a core point when at least minPts neighbours sit within radius eps. Core points that reach each other chain into a cluster of any shape; points that never reach the threshold are labelled noise, which makes it a usable anomaly detector.

• Starting rule: minPts about 2x the feature count; tune eps from the knee of a k-distance plot.
• Both knobs live in your data's units, so scale first.
• eps is a single global radius, so clusters at different densities cannot both work.
• Also loses in high dimensions (distances flatten) and when every point must get a label.

Recall what PCA actually computes

New orthogonal axes ordered by variance; keep the first few and project. Eigenvectors of the covariance matrix are the directions, eigenvalues are the variance along them, and eigenvalue / sum(eigenvalues) is the explained variance ratio you report.

• Components are weighted blends of every original feature, so you can no longer say "the model reacts to income".
• Linear only - curved or clustered structure survives poorly.
• Variance is not usefulness. The direction that separates two classes can be low-variance, and PCA will happily discard it.

Standardize before PCA (with one exception)

Variance carries units, so income in currency has variance millions of times larger than age in years, and PC1 points almost exactly along income. Standardizing means running PCA on the correlation matrix instead of the raw covariance matrix. Fit the scaler on training rows only.

Exception: when every feature already shares one unit (pixels, spectra, same-scale sensors), relative variance is meaningful. Forcing unit variance there lifts quiet noisy channels up to the informative ones.

Decide how many components to keep

Kaiser's rule (eigenvalue > 1 on standardized data) usually keeps too many.

Relate SVD to PCA

X = U S V', singular values sorted, so truncating after k terms is the best rank-k approximation under squared error. On a column-centered X: rows of V' are the principal directions, s_i^2 / (n-1) is the variance of component i, and the scores are U S.

Libraries compute PCA this way because forming the covariance matrix squares the condition number and needs a d-by-d allocation. Randomized truncated SVD returns the top k in roughly O(ndk) without ever building it.

Centering is the whole difference. Truncated SVD on raw sparse counts (centering would destroy sparsity) is latent semantic analysis, not PCA.

Use t-SNE and UMAP for looking only

A nonlinear embedding fitted to one dataset: it matches pairwise neighbour probabilities, preserving local neighbourhoods and sacrificing global structure.

• Distances between well-separated blobs are meaningless, and so are cluster sizes.
• Perplexity and the random seed change the layout. Two runs, two different maps.
• It is not a transform - classic t-SNE cannot embed a new point, so you cannot fit on train and apply to test. Feeding the coordinates into a model leaks and will not generalize.

Use PCA for features; keep t-SNE/UMAP for looking at data.

Recall the curse of dimensionality

Space grows exponentially with features, so a fixed dataset gets sparse fast - holding density steady would need exponentially more rows per feature added. Consequences: distance-based methods lose contrast, and with enough columns one will look predictive by chance, so trees and linear models start fitting noise. More features are not free; feature selection, regularization or PCA usually beats handing the model every column.

• Any preprocessing fit outside the CV fold leaks: scalers, PCA, resamplers, target encoding.
• Tuning and early stopping on the same split. Use a third holdout or nested folds.
• Reporting the CV score at the chosen hyperparameter as the final number - it is optimistic by construction.
• Random splits on time-ordered or repeated-entity data. This breaks OOB estimates too; use grouped or time-based splits.
• Reading a logistic coefficient as a probability change, or comparing raw coefficients across differently-scaled features.
• Trusting Lasso's selected feature set, Naive Bayes probabilities, the elbow plot, or a single K-Means run.