Data Preparation & Feature Engineering.
Cheat sheet.

Order of operations, and the one rule that matters:

1. Before the split: dedupe, drop corrupt rows, fix units, apply row filters. These change what a row is.
2. Split. Time-ordered if the problem is temporal; grouped if one entity has many rows.
3. After the split, fitted on train only: impute, scale, encode, target-encode, select, PCA.
4. Engineer every feature as of a cutoff timestamp: only what was knowable at prediction time.
5. Wrap 3-4 in a Pipeline so cross-validation refits per fold.

Anything that learns a number from data - a mean, a median, a category map, a top-k feature list, a clip threshold - is fitted on training rows and merely applied everywhere else.

Check what your model actually needs

Identify which kind of leak you have

Overfitting vs leakage: overfit = high train, low validation, caught by the process you already run. Leak = high train, high validation, collapses in production, nothing warns you.

Hunt a suspiciously good score

• Ablate: drop the top-importance feature, retrain. If one feature carries everything, check when that column is populated.
• Re-split by time: train on the past, score the future. Leakage usually collapses here.
• Look for duplicate or near-duplicate rows spanning the split.
• Look for IDs that encode the target (a record number that rises with admission date predicts mortality).
• Then accept the boring case: some labels really are near-deterministic.

Fix target encoding leakage

Each row helps compute its own feature value; a category seen once encodes to exactly that row's label.

# out-of-fold: encode each fold from the other folds' statistics
# smoothing toward the global mean, m ~ 10-100
enc = (n * cat_mean + m * global_mean) / (n + m)

Unseen category at serving falls back to the global mean. The serving map comes from training rows only. Noise injection helps less than smoothing and costs signal.

Know what a Pipeline does not fix

pipe = Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)   # scaler refits inside every fold

Covers: transformer parameters learned per fold. Does not cover a leaky column built upstream of X, duplicate rows across the split, a label hiding inside a feature, or group structure (use GroupKFold).

Keep time-series features honest

• Trailing windows only. No centered rolling, no bfill, no interpolation across the cutoff.
• Fit scaler and imputer on the past slice, then transform forward.
• Join on availability time, not event time. A restated revenue figure or a chargeback posted three days late sits in the table today but was not knowable then.
• One global groupby().mean() during a backfill poisons every row at once.

Pick an encoding

Store the mapping as a fitted object, not ad-hoc code. Serving must emit the same columns in the same order or the model silently scores garbage.

Count what one-hot costs you

df["color"].nunique()                        # 3
pd.get_dummies(df, columns=["color"]).shape  # (1000, 12), was (1000, 10)

Width grows with cardinality, not row count: 3 columns of 100 levels adds 300 columns, over 99% zeros. Trees suffer most - a level spread across 100 binary columns needs depth to isolate. Past a few dozen levels, switch encodings.

Replace a high-cardinality label with what it means

For 50,000 cities, drop the raw label and encode the properties: latitude and longitude, median income, urban density, distance to the nearest hub. Those generalize to cities the model never saw; a per-city statistic never will.

Handle a category the model never saw

OneHotEncoder(handle_unknown="ignore")   # all-zeros row instead of raising

An all-zeros row is safe but quiet - fine for a rare new value, misleading if a whole region launched. Target encoding falls back to the global prior. The stronger fix: bucket rare levels into an explicit "other" at training time and map unknowns there. Log the unknown rate; a rising rate is an upstream schema change, not a new city.

Decide whether to drop a dummy level

Drop one level only for an unregularized linear model with an intercept - the dummies sum to the intercept column, so the normal equations go singular and standard errors are nonsense. Ridge, lasso and trees: keep every level. Dropping there just hides a level behind an all-zeros pattern.

Decide whether you need it

Needed wherever magnitudes are compared across features: KNN, k-means, SVM, PCA, ridge, lasso, neural nets. Not needed for tree ensembles - any monotonic rescaling moves the split point with the data, giving an identical tree.

The failure mode is silent. Nothing errors; the model just ignores your small-scale features.

Pick a scaler

Neither min-max nor standardization changes skew. If you wanted symmetry you need a log or power transform.

Watch min-max in production

Values above the training max land above 1 and nothing raises. clip=True saturates instead of extrapolating. Monotone features - account age, lifetime spend, cumulative orders - cross the training max eventually, so accuracy decays on a schedule. Emit the out-of-range fraction as a monitoring signal and retrain when it climbs.

Work the decision in order

1. Count nulls per column and per row. Whole blocks usually mean a broken join or a collection outage, not real absence.
2. Ask why: not-applicable, sensor dropout, and refused-to-answer need different fixes.
3. Choose: drop rows (few), drop the column (mostly empty and weak), impute, or use a GBM that handles nulls natively.
4. Add a col_missing flag when the absence itself is signal (blank income on a loan form).
5. Check what serving does with nulls. Prod filling zero while training filled the median is a mismatch nobody notices.

Pick an imputation

Any constant fill shrinks variance and weakens correlations, and leaves a spike no real distribution has. Treat downstream confidence numbers with suspicion.

Know what MCAR / MAR / MNAR change

The table cannot tell MAR from MNAR - only domain knowledge can. Keep a missing-indicator column either way.

Decide impute vs drop on a 30%-missing column

The percentage is not the decision. Ask: is the observed part predictive, is the field actually populated in production, and is the missingness concentrated in old rows (then the fix is a date filter, not an imputer). Practical default: keep it, add a present/absent flag, then ablate - train with and without, compare on held-out data, keep the winner.

Know what smart imputers cost

KNN imputation needs scaled features, chokes on mixed types, and scans the training set at transform time (roughly O(n^2) in rows). Iterative imputation fits many models per cycle and may not converge. Both learn from the whole table, so fit on train only. Against tree models, median plus an indicator often ties.

Detect them

Trees mostly shrug (they split on order). KNN, k-means and least-squares regression suffer, because squared terms chase the extreme.

Decide what to do with one

• Recording error (age 999, wrong unit, duplicated quantity): fix it or drop the row.
• Genuine extreme: use the lightest tool - clip at a percentile, log-transform, or switch to Huber / quantile loss.

Fit the clip threshold on training rows and reapply it unchanged at serving, or production values land outside the range you trained on. Never delete rows just because the target is extreme - that flatters validation and blinds the model to the cases that cost money.

Scale a heavy-tailed column

Min-max maps the extreme to 1 and crushes everything else near 0. Standardization is milder but its mean and sd are still dragged. Use RobustScaler (median, IQR), or a quantile transform if you need a forced shape - accepting that it flattens the difference between a large order and a huge one. Decide first whether the tail is an error or the thing you are predicting.

Prefer a ratio over a product

size / rooms gives average room size, separating a 2,000 sq ft three-bed from a 2,000 sq ft eight-bed. size * rooms has no physical meaning and mostly re-encodes size. A linear model cannot form a ratio at all; a tree needs many axis-aligned splits to trace the same diagonal.

Guard the denominator - rooms == 0 yields inf, which quietly poisons the scaler. And skip blanket PolynomialFeatures: degree 2 on 50 columns produces over a thousand collinear features.

Turn a timestamp into features

A raw timestamp is nearly useless. Extract hour, weekday, is_weekend, days since last event - and convert to the user's local time zone first, since a 3am purchase is a signal that a shared time zone hides.

df["hr_sin"] = np.sin(2*np.pi*df.hour/24)
df["hr_cos"] = np.cos(2*np.pi*df.hour/24)   # 23:00 now sits next to 00:00

Aggregate a history without leaking

Pick an observation cutoff. Build features only from rows strictly before it; define the label from a window after it. Exclude the current row from its own group aggregate - a per-customer average order value that counts the current order leaks it back. Every window is trailing and keyed on as-of time.

Engineer churn features from purchase history

RFM is the backbone: days since last order, order count, average basket. Levels are weak; trends carry more - orders in the last 90 days versus the prior 90, and the current gap versus that customer's own usual gap. Beyond RFM: category spread, discount dependence, failed payments, support contacts.

If churn means "no purchase in 90 days", days_since_last_order nearly reproduces the label. Expect it to dominate and check the model still works without it.

Engineer fraud velocity features

Counts and sums over trailing windows per card, device, IP and merchant. Pair a 1-minute, 1-hour and 24-hour window - card testing shows up in the short one, account takeover in the long one. Add deltas against the entity's own baseline: seconds since last transaction, hours since a password or address change, implied travel speed between consecutive locations.

The offline pandas window and the online streaming counter must produce identical values. When they drift, the model degrades quietly and nobody notices.

Choose a selection family

With regularized linear models or boosted trees, selecting nothing often matches selecting carefully. Selection earns its keep when features cost money to collect, latency matters, or a human must read the model. Fit the selector on training rows only - scoring against the target on all rows is leakage.

Read a lasso zero correctly

A zero coefficient means a correlated twin won, not that the feature is useless - and which twin survives flips between seeds. Scale inputs first, since the penalty compares raw coefficient magnitudes. Use elastic net to keep correlated groups together, and stability selection (refit on bootstraps, keep features chosen in most runs) when a downstream team needs a fixed column list.

Use correlation without being fooled

Pearson sees straight lines only, so a u-shaped feature scores near zero and is still predictive. Spearman catches monotone curves, mutual information catches general dependence. Univariate cutoffs also discard features that only work in pairs.

corr(feature, target) = 0.97 is usually a disguised copy of the label, not a gift. corr(f1, f2) = 0.99 means one signal, not two.

Choose selection vs extraction

Selection keeps original columns, so a dropped column can stop being collected - fewer joins, cheaper pipeline. A principal component depends on every original input, so serving cost stays; you bought density, not simplicity. Extraction wins on many correlated weak features (pixels, text) and loses when you must explain a decision or debug a drifting input. PCA is unsupervised: its largest variance direction can be pure noise for your target.

Select by label vs by position

With an integer index, df.loc[2] is the row labeled 2, not the third row. On a default RangeIndex the two look identical - that is where the confusion starts.

Fix SettingWithCopyWarning

df[df.qty > 0]["price"] = 0        # writes to a temporary; silently lost
df.loc[df.qty > 0, "price"] = 0    # one setitem; works

If you meant a separate frame, say so with .copy() on the slice. Never silence it with chained_assignment = None - the failure it hides is a write that vanishes without a trace.

Know view vs copy (and Copy-on-Write)

A view shares the parent's buffer, a copy owns memory, and pandas does not promise which you get - it depends on dtypes and block layout, so the same line works on one frame and silently no-ops on another. pandas 3.0 Copy-on-Write makes every derived object behave like an independent copy (buffers shared until something writes), so chained assignment now fails loudly and inplace=True on a chained expression is a no-op.

df.copy() is deep by default: a 4 GB frame becomes 8 GB while both live. Deep means the buffers, not the Python objects - a list inside an object column is still shared. Copy once at a function boundary, not defensively in a loop.

Aggregate several columns in one pass

g = df.groupby("user_id").agg(
    spend=("amount", "sum"),
    orders=("amount", "size"),
    last=("ts", "max"),
)   # flat column names - no MultiIndex to flatten

Use string names like "sum" over lambdas: strings hit compiled paths, a lambda runs Python once per group (an order of magnitude on many small groups). size counts rows, count skips nulls - they disagree exactly where the data is dirty.

Add a group statistic back as a column

df["dev"] = df["x"] - df.groupby("k")["x"].transform("mean")   # length matches df
df.groupby("k")["x"].apply(lambda s: s.mean())                 # one row per group

transform must return group-length output, so it aligns straight back onto the index. apply is unconstrained and infers the shape from whatever you returned - a function that gives a scalar for some groups and a Series for others produces results that change with the data.

Keep the group keys as ordinary columns

df.groupby("city", as_index=False).mean(). Note dropna=True is the default, so rows with a null key disappear silently. df.groupby("city") itself computes nothing until you call an aggregation.

Pick merge vs concat vs join

concat happily produces duplicate index labels, which break later lookups. Pass ignore_index=True when stacking.

Pick a join type

• inner - keys present in both; unmatched rows vanish and row counts shrink quietly.
• left - keep every left row, NaN on the right. The usual choice when enriching a fact table.
• right - the mirror image.
• outer - keep everything, pad both directions.

A key repeated on both sides produces the cross product: 10 x 10 becomes 100 rows. Pass validate="one_to_many" so pandas raises instead.

Merge on multiple columns or the index

a.merge(b, on=["user_id", "date"])                          # both must match
a.merge(b, left_on=["uid", "dt"], right_on=["user", "day"]) # paired by position
a.merge(b, left_index=True, right_on="uid")                 # mix freely

A dtype mismatch is the killer: an int64 key never matches a string key, and you get zero rows with no error.

Stop _x / _y columns appearing

Overlapping non-key column names are silently renamed, so price becomes price_x and price_y. Project the right frame down to key plus wanted columns before merging, or pass explicit suffixes=("_old", "_new") so the names mean something downstream.

Speed up a merge between huge frames

Cut before joining: select only needed columns and filter rows, since merge copies every carried column into the result. Make key dtypes identical, and factorize strings to integer codes or a shared categorical to avoid the slow object path. Sorting both sides and joining on the index enables a merge-join instead of a hash build. pandas merge is single-threaded and needs several times the frame size in transient memory - past that, partition by key hash and merge bucket by bucket, or hand the join to DuckDB or Polars.

Replace loops with vectorized expressions

df["total"] = df.qty * df.price               # one C loop over the column
# [r.qty * r.price for _, r in df.iterrows()] # a Python object per row: 10-100x slower

Both are O(n); vectorizing shrinks the constant factor. The tradeoff is memory - every intermediate expression allocates a full array.

Choose map, apply, or something vectorized

Avoid the silent broadcast blow-up

x = np.arange(100)                # (100,)
(x - x.reshape(100, 1)).shape     # (100, 100) - 10,000 cells, not elementwise
(x - x).shape                     # (100,) - what you meant

NumPy compares shapes from the right; dimensions must be equal or 1, and missing leading dimensions count as 1. Both greater than 1 and unequal raises a clear error - the dangerous case is the one that succeeds. Use keepdims=True on reductions so orientation stays explicit.

Everything below stores a learned parameter. Fit it on the training split, reuse it unchanged on validation, test and production:

• Scaler min/max, mean/sd, median/IQR
• Imputer fill values, including any hand-computed global or group median
• Encoder category lists, column order, and the rare-level "other" bucket
• Target-encoding means and smoothing counts
• Outlier clip thresholds
• Feature selector column list, PCA components, KNN imputer neighbor pool
• Log the unknown-category rate and the out-of-scaler-range fraction; both climbing means retrain