Model Evaluation & Validation.
Cheat sheet.

• Every classification metric is built from the four confusion-matrix cells. When a number looks odd, go back to the cells.
• Pick the metric from the decision the model triggers, not from the algorithm.
• Pick the split from how your rows depend on each other: iid -> stratified k-fold, repeated entities -> group split, time -> forward chaining.
• The threshold is a separate tunable decision from the model. Train once, threshold later.
• The test set is a measuring instrument, not a tuning signal.

Read a confusion matrix

from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
# [[tn fp]      rows = actual, cols = predicted
#  [fn tp]]     the four cells sum to len(y_true)

Other libraries transpose this. Check axis order before reading someone else's matrix.

Compute the core metrics by hand

Precision, recall and F1 ignore TN entirely. P=1.0 with R=0.1 gives F1 = 0.18, not 0.55.

Get every per-class number at once

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred, digits=3))
# per-class precision/recall/f1/support + macro and weighted averages

Decide whether precision or recall wins

Ask what happens after a positive prediction.

Average multi-class scores correctly

Everything is one-vs-rest per class, plus a K x K confusion matrix. The per-class table tells you something; the averages are where meaning gets lost.

AUC: multi_class="ovr" is the default; "ovo" is steadier under skewed class sizes.

Look up the sklearn scoring strings

Regression scorers are negated so that higher is always better. Flip the sign before you quote them.

Know when accuracy is lying

State the majority-class baseline before the score. With 1% positives, "always negative" scores 99%. If accuracy barely beats the baseline, the model learned the prior and nothing else. Accuracy also moves with prevalence, so month-over-month comparisons break.

Replace accuracy

1. Per-class precision and recall for the rare class, first, always.
2. One summary: average precision (PR AUC), MCC, or balanced accuracy.
3. One operational number: recall at fixed precision, or precision@k where k is review capacity.

Prefer PR curves to ROC when positives are rare

FPR has a huge denominator when negatives dominate, so thousands of false alarms barely dent the ROC curve. Average precision reacts sharply to the same alarms. PR baseline = the positive rate; ROC baseline = 0.5 regardless of prevalence.

Diagnose a model that predicts one class for everything

This is not overfitting; the model learned nothing input-specific.

• Check recall on the rare class: it will be 0, which accuracy hides.
• Check the predicted probabilities: clustered tight around the base rate means no usable signal.
• Usual causes: imbalance plus an accuracy-shaped objective, learning rate too high, zeroed or dead features.
• Fix features or class_weight, not regularization.

Set the threshold from error costs

cost_fp, cost_fn = 1, 10
threshold = cost_fp / (cost_fp + cost_fn)   # 0.09 -> act on scores above this

Assumes calibrated probabilities and a stable base rate. Boosted trees output squashed scores; calibrate first.

Set the threshold from capacity

If reviewers clear 500 alerts a day, pick the threshold that emits 500 alerts and spend the rest of the effort on ranking. Capacity overrules the cost arithmetic in most production systems.

Pick a threshold off the curve

from sklearn.metrics import precision_recall_curve
p, r, thr = precision_recall_curve(y_true, y_score)
# len(thr) == len(p) - 1 ; p[-1]=1 and r[-1]=0 are sentinels with no threshold
best = thr[(2*p*r/(p+r+1e-12))[:-1].argmax()]

The threshold is a tuned parameter. Choosing it on validation and reporting that same set's cost is optimistic. Keep a fresh slice for the final number, and recheck the operating point as the base rate drifts.

Precision and recall share the threshold, so tuning only slides you along the existing curve. More of both at once needs a better model: sharper features, more data, or a different loss.

Read a ROC curve

y = TPR, x = FPR, one point per threshold. Diagonal = random. Bottom-left = flag nothing, top-right = flag everything. Bulging toward the top-left = good separation. Pick the largest FPR you can absorb, read up, take that cutoff.

Interpret an AUC number

AUC = P(a random positive scores above a random negative), ties counting half.

It depends only on the ordering, so any monotone rescaling leaves it unchanged. That means AUC says nothing about calibration and nothing about which threshold to ship.

Compute AUC from ranks (Mann-Whitney U)

# ranks 1..n after sorting by score; P = #positives, N = #negatives
auc = (sum_positive_ranks - P * (P + 1) / 2) / (P * N)   # O(n log n), no curve needed

Decide which of two classifiers is better

1. Dominance? One curve above the other at every FPR -> better at any threshold, done.
2. Curves cross? Compare only inside the band you will actually operate in (e.g. FPR < 1%). Global AUC averages over regions you will never run in, so higher AUC can be worse where you live.
3. Noise? A 0.005 gap on 200 positives is not a result. Use bootstrap bands or compare AUCs across CV folds.

Check whether probabilities mean anything

from sklearn.calibration import calibration_curve
prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=10, strategy="quantile")
# calibrated == points fall on the diagonal: ~70% of the 0.7 bin are positive

Fix miscalibration

from sklearn.calibration import CalibratedClassifierCV
cal = CalibratedClassifierCV(clf, method="isotonic", cv=5)   # "sigmoid" = Platt

sigmoid for small data, isotonic for roughly 1k+ rows (it overfits below that). Fit the correction on held-out data, never the training folds.

Know when it matters

Needed as soon as a score feeds arithmetic: expected loss = probability x amount, cost-based thresholds, blending several models' scores. Ranking-only uses do not care, which is why AUC is identical after any recalibration.

A model that predicts the base rate for everyone is perfectly calibrated and useless. Pair calibration error with a ranking metric, or use Brier score, which penalizes both failures.

Pick one

RSS = squared gaps to your predictions. TSS = squared gaps to the target's mean, i.e. the error of a mean-only model.

Know what squaring buys and costs

a = [10, 0, 0, 0]          # MAE 2.5, MSE 25.0
b = [2.5, 2.5, 2.5, 2.5]   # MAE 2.5, MSE 6.25

Same total absolute error, 4x the MSE. RMSE chases outliers, and so does any model trained to minimize it. When extreme rows are noise rather than signal, that pull drags predictions off the bulk of the data.

Avoid the MAPE traps

• Undefined at y = 0, and explodes for near-zero actuals.
• Asymmetric: over-prediction can score past 100% error, under-prediction is capped at 100%. On slow-moving inventory this quietly biases forecasts low.

Avoid the R-squared traps

• In-sample R-squared can only rise when you add a variable. Enough junk columns reach 1.0.
• Adjusted R-squared prices that in, but it is still in-sample and its penalty is mild. For predictive work, compare held-out error instead.
• On held-out data R-squared can go negative: worse than predicting the mean. That is a real failure signal, not a bug.

Diagnose from two numbers

Size the gap against the noise in the metric: a two-point drop on a small validation set can be luck; 0.99 -> 0.72 is not. Confirm across folds before acting.

Read a complexity or learning curve

Sweep complexity: training error falls the whole way, validation error drops, bottoms out, then climbs. The bottom of that U is the complexity you want. Curve level = bias; gap between curves = variance.

The clean bias/variance/noise split holds exactly for squared error, not accuracy or log loss. Very large models can break the U (validation error falls again past zero training error). And a small gap can just mean the validation split is too easy or shares rows with training.

Run k-fold and report it

from sklearn.model_selection import cross_val_score, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
s = cross_val_score(pipe, X, y, cv=cv, scoring="f1", n_jobs=-1)
print(f"{s.mean():.3f} +/- {s.std():.3f}")   # e.g. 0.840 +/- 0.030 over 5 folds

Report the mean with the spread, never the best fold. A wide spread is itself the finding. The mean estimates a recipe, not one trained artifact. k = 5 or 10 for tabular work; cost is k fits.

Pick the right splitter

Stratification costs nothing and removes a real failure mode: 30 positives over 5 random folds can leave one fold with zero positives, making recall undefined and the mean garbage.

Avoid setting k too high

Three problems at once: k fits of compute; training sets that overlap almost completely, so fold scores are correlated and averaging barely cuts variance; and folds too small for stable per-fold precision or recall. Leave-one-out earns its cost only when n is tiny and the model is cheap, or when a closed form exists (OLS, ridge). It cannot be stratified.

Cross-validate time-series data

from sklearn.model_selection import TimeSeriesSplit
for tr, te in TimeSeriesSplit(n_splits=5, gap=30).split(X):
    ...   # fold 1: train months 1-3, test month 4; fold 2: train 1-4, test 5

Random folds train on the future and predict the past, and autocorrelation makes neighbouring rows give the answer away. Set gap when labels take time to mature (a 30-day default flag). Expanding window uses all history; a sliding window forgets old regimes and wins when behavior shifts.

Build the model you actually ship

Refit one model on all the data using the settings CV chose. Fold models were disposable estimators of performance. The CV score still describes the shipped model well, since the refit sees slightly more data.

Two rough edges: anything tuned per fold (early-stopping epoch count, threshold) has no single value to refit with, so average it or keep a small held-out slice for that decision. Averaging the fold models into an ensemble often scores higher, at k times inference cost.

Use nested CV (and know when not to)

Inner loop searches hyperparameters on the training part of each outer fold; the outer loop scores that winner on rows the search never saw. It estimates the whole tuning procedure, and hands you no model to ship.

Need it when data is small, many configurations were searched, and the reported number must hold up, especially when comparing model families. Cost multiplies: 5 outer x 5 inner x 100 configs = 2500 fits. With plenty of data, a plain three-way split is cheaper and just as honest. Whatever you nest must cover every fitted step, including feature selection and threshold choice.

Search a grid with CV in one call

from sklearn.model_selection import GridSearchCV
grid = {"max_depth": [3, 5, 10], "min_samples_leaf": [1, 5]}
search = GridSearchCV(tree, grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(X, y)            # 3 * 2 * 5 = 30 fits
search.best_params_         # winner
search.best_estimator_      # refit=True by default: already refit on all data
search.best_score_          # OPTIMISTIC - same folds chose it and graded it

Fit count multiplies. Adding one more parameter with five values takes 30 fits to 150.

Choose grid versus random search

Random usually wins on equal budget: most hyperparameters barely matter and one or two dominate, so a 5x5 grid spends only 5 distinct values on the parameter that counts while 25 random draws spend 25. Grid still wins for small discrete spaces (3 kernels x 4 depths) and when you want a readable table of one parameter's behavior. Sample continuous ranges log-uniform either way.

Tune on a limited compute budget

• HalvingRandomSearchCV: train many configs briefly, keep the best half, repeat with longer runs.
• Coarse wide log-scale sweep first, then resample tightly where good scores cluster.
• Sweep on a stratified subsample with 3 folds, then confirm the finalist properly.
• Freeze parameters that never move the score; search the two that do.
• Bayesian search loses to random below roughly 20 trials; it never learns enough to steer.

Audit for leakage

• Per column: does this value exist at decision time? closure_reason, refund_date, anything written after the outcome.
• Preprocessing before splitting: scaling, imputing, feature selection, target encoding or SMOTE fit on the full dataset pushes test statistics into training. Wrap every step in a Pipeline so it refits inside each fold.
• Non-independent rows: several rows per user, patient or sensor window put near-duplicates on both sides of a fold. Split by that key.
• Time ordering ignored: use forward chaining, plus a final holdout taken from the newest data.

pipe = Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression())])
cross_val_score(pipe, X, y, cv=cv)   # scaler refits inside every fold

A score that is too good plus one feature dominating importances is leakage until proven otherwise.

• Quote precision and recall together, plus how many predictions produced them.
• Always state the majority-class baseline next to accuracy. "94%" alone is close to meaningless.
• Trust the held-out number; treat the training number as a diagnostic only.
• Every time you look at the test set and change something, it becomes another validation set.
• One headline metric plus a few guardrails. Report it per segment too, because a strong average hides one segment the model fails.
• Weight by money where money is skewed: catching 90% of fraud cases can still miss most of the dollars, and count-based F1 can improve while losses grow.
• Tie the metric to the capacity that consumes the positives (recall in the top k, alerts per analyst hour).
• Costs and base rates drift, so a metric and threshold tuned once quietly decay. Re-measure on recent data.