Machine Learning Fundamentals.
Cheat sheet.

Ordinary code: you write the logic, the computer returns answers. Machine learning: you supply the data and the answers, and training returns the logic - a model, meaning a set of numbers fitted to past examples.

Pick the learning paradigm

Semi-supervised recipe (pseudo-labeling): train on the labeled set -> predict the rest -> keep confident guesses -> retrain. Reach for it when labels are expensive and raw data is nearly free (scans, legal docs, call audio).

Pseudo-labels bake early mistakes into later training. If the unlabeled pool is a different distribution, plain supervised learning on the small clean set wins.

Reinforcement learning needs millions of episodes, so it needs a cheap simulator. Use it only when decisions are sequential and each choice changes what comes next (routing, control, restocking).

Tell classification from regression

Decide: hand-written rules or a model

Split a dataset

60 / 20 / 20 train / validation / test is the default. With millions of rows, 1 percent each for validation and test is plenty.

Every peek at the test set spends some of its value. Ten rounds of "check, tweak, check" tunes you to that sample and the number stops predicting production.

Split time-ordered or grouped data

train = df[df.event_date <  "2026-01-01"]
test  = df[df.event_date >= "2026-01-01"]   # random split would leak the future

Group-split by entity too: the same customer (or a near-duplicate row) on both sides leaks quietly, even with a correct time cut.

Tell overfitting from underfitting

The fixes point in opposite directions. Adding capacity to an overfit model makes it worse; simplifying an underfit one does the same.

Tell high bias from high variance

Dartboard: bias is aiming at the wrong spot, variance is a shaky hand.

Recall the error decomposition

Expected squared error = bias² + variance + irreducible noise. Bias² is how far your average prediction sits from the truth; variance is how much it moves between retrains; noise is in the label and no model removes it.

This split is a property of squared error, not a law of learning. Under 0-1 classification loss it is far messier - variance sometimes knocks a prediction back onto the right side of the boundary.

Decide whether to buy data or change the model

• Wide misses that every retrain agrees on -> bias -> add capacity or features.
• Predictions that swing between retrains -> variance -> more rows or more constraints.
• Both small and error still high -> you hit the noise floor; only new signal moves it.

Cut overfitting, in order of payoff

1. More training data, or cheap augmentation of what you have.
2. Fewer / simpler features, especially high-cardinality identifiers that let the model memorize rows.
3. A smaller model: shallower trees, fewer parameters, lower polynomial degree.
4. Regularization or early stopping.
5. Ensembling - average several noisy fits into one steadier prediction.

First rule out leakage and a train/production mismatch. No penalty term fixes a column that encodes the label, and a simpler model just fails more politely on the wrong distribution.

Cut bias

Richer features and interaction terms, a more flexible model family, weaker regularization, train longer, or boosting (each new model fits what the earlier ones got wrong).

More rows do nothing for bias. Neither does averaging near-identical models. A model that cannot bend keeps missing the same way with ten times the data.

Lower variance without raising bias

Only two honest routes: more data, and averaging independent fits (bagging; random forests go further by decorrelating trees with random feature subsets). Regularization does not qualify - it buys variance with bias.

Variance only falls toward the error the members share, so twenty correlated models buy almost nothing. k models also cost k times the memory and latency. Bagging assumes exchangeable rows; on time-ordered or grouped data the gain is fake.

Tune regularization strength

Tune the penalty against held-out data, never training error - training error only climbs as the penalty grows. Turning it up on a model that already underfits just makes it worse.

Handle missing values

Find out why first. Random sensor dropouts differ from an income field left blank by low earners - in the second case the gap itself is signal.

df["income_missing"] = df.income.isna().astype(int)   # keep the signal
df["income"] = df.income.fillna(train_income_median)  # median from TRAIN only

Three moves: drop rows (fine when few and random), drop the column, or impute (median for numeric, mode for categorical).

Handle outliers

Kill duplicates

Give duplicates their own pass. The same row landing in both train and test inflates your score, and repeated rows overweight whatever they happen to contain.

Keep quality high in the pipeline, not in a one-off cleanup

Assert a schema and value ranges at ingestion and fail loudly instead of coercing silently. Record each field's source. Rerun the checks every time new data lands. Automated checks catch shape, not meaning - read a random sample of rows by hand against the source system to catch the field that changed units or the join that duplicated every customer.

Do not clean until the data looks nice. Training data must match what arrives at serving time, warts included. Fix genuine corruption; keep the ugliness that is real.

Get labels you can trust

Write the label definition, the edge cases, and worked examples before hiring more labelers. Double-label a batch with two people and measure agreement; a low number is a specification bug, not something majority voting fixes. For genuinely ambiguous items, route to an expert, add an explicit unclear bucket, or keep the soft/split vote. Always hold back a clean hand-checked slice for measuring.

Spot leakage in a feature

Leakage is information in the training data that will not exist at prediction time. Ask one question of every column: was this value knowable before the moment of prediction? If no, it cannot be a feature.

Classic tell: a loan-default model handed collections_agency_assigned. That field is only filled in after the default.

Fit preprocessing without leaking

Split first, then fit every transformation on training rows alone.

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), model)
pipe.fit(X_tr, y_tr)     # imputer + scaler see training rows only
pipe.score(X_te, y_te)   # transforms applied, never refitted

Leaky if fitted before the split: scalers, imputers, category encodings, feature selection, PCA. Worst offenders are target-based: mean-encoding a category by its label, or an aggregate computed over the full history instead of point-in-time.

Tell leakage from ordinary overfitting

Suspiciously perfect results are the tell. Nobody debugs a model that looks great, which is exactly how leakage reaches production.

Hunt leakage in an existing pipeline

• Score audit - a metric far above what the problem allows is a bug report, not a win.
• Time audit - for each column, when did its value become knowable? Refund flags, cancellation reasons, status fields are suspects.
• Ablation - drop top features one at a time; if one column's removal collapses the score, read its source.
• Split swap - retrain with a time-ordered split; a big drop versus a random split means future rows were teaching the model.
• Identity check - same customer or near-duplicate rows on both sides of the split.
• Shadow run - score live traffic; offline >> online means the offline features were not available at decision time.

Fix the objective before touching the rows

A rare class only hurts when it is also poorly separated. What breaks first is usually the loss, which happily predicts the majority class forever.

model = LogisticRegression(class_weight="balanced")   # or scale_pos_weight in XGBoost
p = model.predict_proba(X)[:, 1]
pred = p > 0.12    # threshold from cost of a miss vs a false alarm, not 0.5

Judge on the rare class (precision/recall/PR-AUC at your operating point), never on overall accuracy.

Choose a resampling method

Resample the training split only - synthesizing before the split leaks. Resampling distorts predicted probabilities, so recalibrate if you use them.

With only a few dozen positives, no reweighting invents signal: collect more, or treat it as anomaly detection.

Decide whether to scale

Distance and weighted-sum models let a salary column in the tens of thousands drown an age column, and gradient descent zigzags on a stretched error surface. Trees split one feature at a time on thresholds, so scaling changes nothing.

Pick a scaler

Standardize (zero mean, unit variance) by default. Min-max to 0-1 is fragile with outliers - one extreme value crushes everything else into a sliver. Use a robust / quantile scaler for heavy tails.

Fit the scaler on training rows and reuse those exact statistics at inference. Recomputing statistics per live batch silently shifts every prediction as traffic drifts.

Choose a first model

Build a dumb baseline first: majority class, last month's value, or the current heuristic. Plenty of projects quietly die when the real model barely beats it.

Start simple to learn the data, not out of ritual. If everyone already knows which family wins, skip the ceremony.

Parametric vs non-parametric

"Non-parametric" does not mean no parameters - it means the count is not fixed in advance.

Run the project in order

1. Frame - which decision does this serve, what is success, what does the current process cost?
2. Get data - collect, join, label, then actually look at the rows.
3. Baseline - a simple rule everything later must beat.
4. Iterate - features, models, held-out checks; stop when gains stop justifying complexity.
5. Ship - serving, latency budget, fallback for when the model is unavailable.
6. Monitor - input drift, outcome decay, retraining planned before it is urgent.

Skip framing and you get an accurate model nobody uses. Skip the baseline and you never learn whether the model was needed.

Frame churn prediction

• Label is a dated event: a cancellation or lapsed renewal; for usage products, an inactivity threshold like no session in 30 days.
• Two windows: how much history you observe, and how far ahead you predict.
• The horizon comes from the intervention, not the data. If a save offer takes two weeks to land, firing three days out is useless.
• Score currently-active accounts on a schedule and rank against the retention team's real outreach capacity.
• Exclude involuntary churn (expired cards) - that is a billing fix and it poisons the labels. Never-activated users need their own model.
• Cost at scale is point-in-time features: one row per user per snapshot multiplies fast. Sample snapshots and precompute aggregates instead of replaying event logs per training run.

Frame fraud detection

• It is ranking plus a threshold inside a hard latency budget - authorization needs a score in tens of milliseconds.
• Features: precomputed profile values plus a few live velocity counters (transactions on this card in the last 5 minutes, distance from the previous merchant).
• Set the operating point from review capacity and two prices: a declined genuine purchase (support call, customer stops using the card) versus a missed fraud (chargeback).
• Labels arrive late - chargebacks land weeks after the transaction, so recent data is only partly labeled and looks deceptively clean.
• Fraud is adversarial: patterns decay in weeks, so retraining cadence matters more than model choice.
• At scale the counters break first. A stale or missing counter turns a strong feature into a constant and the score degrades with no error raised.

• A score far better than the problem allows is leakage until proven otherwise.
• Reporting a number from rows the model was tuned on. Validation absorbs tuning; test is spent once.
• Fitting a scaler, imputer, or encoder before the split.
• Random-splitting time-ordered data, or letting one customer sit on both sides.
• Deleting outliers on sight - you teach the model a world without rare events, which is where the expensive mistakes live.
• Imputing a not-random gap without a was_missing flag.
• Judging an imbalanced model on accuracy, or leaving the threshold at 0.5.
• Adding rows to fix bias, or adding capacity to fix variance.
• Scaling features for a tree model and calling it progress.
• No baseline, so nobody can say whether the model was needed at all.