Leakage means your training data contains information that would not exist at prediction time, or that quietly encodes the answer. The model learns that shortcut, because it is the easiest signal available.

The damage is that your offline numbers become fiction. Validation accuracy looks excellent, so you stop investigating and ship. In production the leaked column is empty, or it arrives after the decision, and performance collapses toward baseline.

It is hard to catch because nothing crashes. The code is correct; the data is wrong. A classic case is predicting fraud using a chargeback_filed column, when chargebacks only get filed after fraud is confirmed.

The cost is time and trust. Weeks of tuning on a leaked feature buy nothing, and business decisions get made on a number that was never true.

Feature engineering is the work of reshaping raw columns into inputs that expose structure the model can actually use. A raw timestamp is nearly useless; hour of day, day of week, and days since last order are not.

On tabular data the effect is large, usually larger than swapping model families. A gradient-boosted tree with well-built ratios and aggregates beats a heavily tuned network fed raw columns. Most real gains on structured problems come from the features.

The exception is domains where the model learns its own representation. For images, audio, and text, deep networks build features internally, so hand-crafted ones add little.

The cost is time and maintenance. Every engineered feature is code that must run identically in training and in serving, or the model sees something different once it is live.

Domain knowledge tells you which combinations of columns mean something, and a blind search over raw columns cannot guess that. In card payments, transaction amount alone is weak. Amount divided by that customer's usual amount is strong.

It also tells you what exists at decision time. Someone who runs the process knows the risk score is written back two days later. That single fact prevents a leaked feature from ever being built.

It flags proxies too. Postcode often stands in for income or ethnicity, which matters for fairness and for stability when a neighborhood changes.

Without it, you brute-force thousands of combinations and hope a selection method sorts them out. That is slower, noisier, and produces features nobody can explain to the team relying on the prediction.

Models need numbers, so every category has to become one without inventing an order that is not there. Two questions decide the method: do the categories have a real ordering, and how many distinct values exist?

Few values and no order means one indicator column per value. Few values with a genuine order means integer codes that preserve it. Many values means grouping the long tail into "other", hashing, target statistics, or a learned embedding.

The model family matters as well. Trees split on integer codes without reading them as magnitudes, so they tolerate compact encodings. Linear and distance-based models read 3 as three times 1.

Whatever you pick, store the mapping as a fitted object rather than ad-hoc code. Serving must produce the same columns in the same order, or the model silently scores garbage.

One-hot encoding replaces a categorical column with one binary column per distinct value; each row carries a single 1. A color column of red, green, and blue becomes three columns.

df["color"].nunique()                        # 3
pd.get_dummies(df, columns=["color"]).shape  # (1000, 12), was (1000, 10)

Dimensionality grows with cardinality, not with row count. One column of k values costs k columns. Three columns of 100 values each add 300, and the resulting matrix is over 99 percent zeros.

That width costs memory and fit time, and it hurts trees in particular. A tree splits one column at a time, so a category spread across 100 binary columns needs depth to isolate. Sparse storage fixes the memory, not the depth. Past a few dozen values, reach for another encoding.

Reach for ordinal encoding when the categories have a genuine order and the model can exploit it. Low, medium, and high severity, education level, t-shirt sizes, and survey ratings all carry order that one-hot throws away.

Encoded as 1, 2, 3, the model can learn a single split at "greater than 2". One-hot would need several splits to express the same idea, and it treats each level as unrelated.

The second case is high cardinality feeding a tree model. Integer codes keep the frame narrow, and trees do not read the integers as magnitudes anyway.

The cost lands on linear and distance-based models. They assume the gap from 1 to 2 equals the gap from 2 to 3. If your order is invented, say country codes, you have handed the model a relationship that does not exist.

Scaling puts numeric features onto comparable ranges so no feature dominates purely because of its units. Salary in the tens of thousands and age in the tens are not comparable until you fix that.

It is essential wherever the algorithm compares magnitudes across features. Distance-based methods like k-nearest neighbors and k-means, support vector machines, principal component analysis, and any model with a regularization penalty all qualify. Neural networks need it too.

The mechanism repeats each time. Distance sums squared differences, so the widest feature owns the result. A penalty applied per coefficient shrinks whichever features happen to use small units. Variance-maximizing methods follow units directly.

Tree-based models do not need it. The failure mode elsewhere is quiet: nothing errors, the model just ignores your small-scale features. Fit the scaler on training rows only.

Normalization usually means min-max scaling: subtract the minimum, divide by the range, landing every value between 0 and 1. Standardization subtracts the mean and divides by the standard deviation, giving mean 0 and spread 1 with no fixed bounds.

The practical split is bounds versus distribution. Min-max guarantees a range, which suits pixel values and network inputs that expect one. Standardization guarantees location and spread, which is what linear models and principal component analysis assume.

Neither changes the shape of the distribution. Standardizing a skewed column leaves it skewed; if you wanted symmetry, you need a log or power transform instead.

Min-max is the fragile one. A single extreme maximum squeezes every other value into a narrow band near zero, and the range you fitted on rarely holds for future data.