Three shapes cover nearly every case: batch scoring, online request-response, and streaming prediction. Batch runs on a schedule and writes predictions into a table or cache. Online serving computes a prediction inside a request, under a latency budget. Streaming scores events as they arrive on a queue and pushes results downstream.

Batch wins when the prediction does not depend on something the user just did. Churn scores, nightly recommendations, lead ranking, and credit pre-approvals all fit. You get cheap hardware, no tail-latency worry, and easy retries when a run fails. Reading a precomputed row is a key lookup, so the serving path stays trivial.

Batch loses the moment freshness matters. A score computed overnight is stale for a session that started at lunchtime. The usual cost of getting this wrong is a recommender that ignores everything the user did in the current session.

Serving is the running system that answers prediction requests. Inference is the single act of computing one prediction from one input. Deployment is the release event that puts a specific model version behind the serving system.

The distinction matters when something breaks. Inference is a math problem you can reproduce in a notebook. Serving is an availability problem with queues, timeouts, autoscaling, and a p99 latency number. Deployment is a change problem with rollback, approval, and a record of who shipped what.

Teams blur the three and then own the wrong thing. A model that scores well offline still needs capacity planning, warm-up, and a health check before it can carry traffic. A perfectly healthy service can also serve a stale model for months. That happens when nobody treated deployment as its own step with its own audit trail.

Model drift is the decay in a model's real-world performance after it ships. The weights never change. The world those weights were fitted to does change, so yesterday's good fit becomes today's mediocre one.

Three things move underneath a frozen model. User behaviour shifts with seasons, prices, and competitors. Upstream systems change, so a field that meant one thing now means another. The model also changes behaviour by acting on it, since users mostly see what it ranked highly.

So a launch-day accuracy of 92 percent is not a property of the model. It is a measurement of one moment. Treating it as permanent is how a fraud model quietly misses a scam pattern invented after training ended. Plan for decay from day one, and budget retraining as a running cost rather than a rescue project.

Experiment tracking is the habit of recording every training run as a durable row, not a terminal log. Each run gets an id, and everything needed to explain its result hangs off that id.

Record at minimum:

• the git commit of the training code, including preprocessing
• the dataset version, or a snapshot of the query that built it
• every hyperparameter and the random seeds
• the environment: library versions and hardware
• metrics on a fixed evaluation set, plus the artifact and its hash

The payoff shows up weeks later. Someone asks why last month's model beat this week's, and the honest answer without tracking is a shrug. With it you diff two rows and find the changed learning rate or the extra week of data. It also ends the classic waste where a strong result cannot be reproduced because nobody recorded the branch.

A model registry is the catalogue of trained models that are candidates for production or already in it. Each entry holds an immutable version, the artifact, its metrics, its stage, and a pointer back to the run that produced it.

Versions matter because a model is a moving binary with no compile-time contract. Two files named fraud_model.pkl can behave completely differently, and neither will complain. Without a version stamped into the serving logs, you cannot say which model produced a specific bad prediction.

The registry also gives deployment something stable to point at. The service asks for the production alias, not a file path on someone's laptop or a bucket key that got overwritten last week. That one layer of indirection turns rollback into a config change instead of a rebuild. It usually pays for itself during the first bad release.

Reproducibility means rebuilding the same model artifact, or a statistically equivalent one, from a recorded starting point. It rests on three axes: the data, the code, and the environment. Miss any one and the rebuild drifts away from the original.

Data means the exact rows and values as they were then, not a table that has since been updated in place. Code means a commit hash, including the preprocessing that lives outside the training script. Environment means pinned library versions, because a minor bump in a numerical library can shift results.

It buys you three concrete things. Debugging a bad prediction needs the model that made it, not a lookalike. Regulated domains need proof of how a decision was produced. And any claim of improvement is empty if the baseline cannot be rerun. Bit-for-bit determinism on GPUs is expensive, so most teams settle for pinned inputs plus fixed seeds.