MLOps & Model Deployment.
Cheat sheet.

Pick batch, online, or streaming

Most "we need streaming" is really "we need one fast feature lookup". Precompute that feature and stay request-response.

Decide what to return when the model is unavailable

• Hard timeout + circuit breaker in front of the call. A slow model is worse than a dead one.
• Identical response schema in both paths; add a degraded: true + reason code.

Tag every degraded response in the log, or the next retrain learns from your own fallbacks.

Order the fallback tiers (fire per failed component, not one global switch)

Log this for every training run

git commit of training and preprocessing code; dataset version or snapshot id; all hyperparameters + seeds; library versions and hardware; metrics on a fixed eval set; artifact + its hash.

Pin a run so it rebuilds in six months

Target metric-equivalence, not bit-exactness (bit-exact GPU determinism costs real training speed).

The one people skip is the environment. A :latest-style tag reproduces next week and fails next year.

Version a dataset (pin bytes, not a filter)

# same query + same cutoff != same rows, once backfills and deletes land
run = {"dataset": "orders", "snapshot": "s_00147", "sha256": "3ab9..."}
# rebuild -> load s_00147, verify hash, retrain -> same AUC within noise

Two workable shapes: immutable snapshot copy + content hash, or an append-only table format with retained snapshots and the snapshot id in the run manifest. Retention must outlive your audit window. Erasure requests delete rows from old snapshots - document which runs are now approximate.

Promote staging -> production in a registry

Promotion is an alias change, not a deploy. The artifact bytes never move.

1. Register artifact after training, attach metrics + lineage.
2. Staging gate: held-out set + a shadow window.
3. Human or policy approval, then flip the alias. Archive the old version, never delete.

If serving pins a hard version, "promotion" changes nothing until someone redeploys.

Make the training job refuse to run

...when dataset hash, commit hash, or image digest is unknown. Record identifiers, not copies. One untracked notebook breaks the whole chain.

Keep training and serving features identical

One definition, two reads: training reads the offline store, serving reads the materialized online value. Neither recomputes the transform locally. The moment someone computes a feature inline in serving code, skew is back and nothing warns you.

Defense: log the exact feature vector used at prediction time, then diff it against the training row.

Use point-in-time (as-of) joins

The offline read must return the value as it stood at the event timestamp. Using today's value leaks the future and inflates offline scores.

Skip the feature store when

One model, one nightly job, or features computed from the request payload alone. A shared transformation library is simpler and just as safe. The store earns its network hop, its uptime risk, and its bill once several models share features across teams.

Debug "offline and online disagree"

Pin one entity + one timestamp and replay end to end. Compare the logged served vector against a recompute - not against a fresh pipeline run; the pipeline is the suspect.

Run this comparison continuously on a small sample, not only when someone complains.

Watch four layers, in arrival order

Never page overnight for a two-point move in a feature histogram. That is how teams learn to ignore the pager.

Pick a drift test

PSI wins because its thresholds behave like an effect size. With enough traffic a p-value fires on everything.

Choose what to monitor (not every column)

Top features by importance; anything from a flaky upstream source; the model's own output distribution (summarises many inputs at once). Slice by country, device, app version - a shift confined to one slice is a pipeline change, not the world moving.

Tell a broken pipeline from concept drift

Rule out the pipeline first - it is far more common and far cheaper to fix. The nasty case is slow rot (a source table gradually stops populating for a growing share of users); catch it with freshness and coverage monitors on the source, not distribution monitors on the feature.

Detect degradation without labels

• Prediction distribution shift - cheapest alarm (avg fraud score halves overnight).
• Input drift weighted by feature importance, not flat across every column.
• Weak labels: human-review a few hundred sampled predictions weekly.
• Proxy outcomes: acceptance rate, click-through, manual override rate - all move before the real label.

Two blind spots: drift without harm (false alarms - tie thresholds to observed impact) and harm without drift (concept drift, which no input monitor can see).

Monitor many models at once

Standardize emission, centralize compute, tier the alerting. One shared logging library emits the same envelope everywhere: model_id, version, sampled feature vector, prediction, latency, request_id. One drift job reads the store for all models, baselines pulled from each model's training snapshot. Monitoring attaches at registration, not as a follow-up ticket.

Cost bites twice: store sketches/histograms and sample rather than keeping full vectors, and watch cardinality - model x version x slice x feature will melt a TSDB.

Give each trigger the response it deserves

The underrated case for a cadence: a pipeline that has not run in five months will fail the day you need it urgently. Shorten the cadence when label latency is long.

Promote a challenger without shipping a worse model

• Re-score both models on the same recent holdout with the same eval code. Comparing a fresh challenger against the champion's stale historical score is the classic bug.
• Default is no promotion. Promote only if the gain exceeds run-to-run noise (measure noise by training the same recipe twice). A tie keeps what is serving.
• Add per-segment no-regression rules; an overall gain can hide a collapse in one country.
• Add prediction-distribution and feature-schema checks - they catch a broken input before quality tests run.
• Full automation is fine. Automatic promotion straight to 100% of traffic is not.

Retrain vs fine-tune

Diagnose "accuracy slid for three months"

1. Confirm the metric is real: label delay, changed denominator, new logging path. Recompute on a fixed cohort.
2. Segment before modeling. Average accuracy often falls because a weak segment grew (new country, new device class, bot wave), not because anything got worse.
3. Only then fix by cause: grown weak segment -> more data or its own treatment; genuine decay -> retrain on a shorter window with a fresh holdout, compared to the incumbent on the same recent data.

Retraining first hides a mix change, and the slide resumes a month later.

Pick a rollout mechanism

Choose from how fast you learn the model is bad: labels in seconds -> a small live slice teaches you within an hour; ground truth in two weeks -> a canary only proves the service does not crash. Then check for irreversible side effects (sent email, declined payment) and feature contracts (a new feature must exist in serving first).

A shadow model must never write anywhere.

Do not confuse canary with A/B

Randomizing an A/B by request is the classic bug: one user sees both models and per-user metrics become meaningless. Canary first for safety, A/B after for value.

Auto-halt a canary on these

Guard every rule with a minimum sample count. Use sequential tests (rechecking a fixed threshold each minute manufactures false alarms). Exclude the opening minutes - cold caches and JIT inflate early tails. Past a few dozen models, learn each baseline from the control arm per rollout instead of hand-setting numbers.

Handle "canary wins on revenue, loses on p99"

1. Is the tail real? Small canary = few p99 samples. Re-measure after warmup, on matched hardware, split by slice.
2. Do upstream callers time out near that tail? Then slow predictions became errors and the lift is measured on survivors only.
3. Convert to one currency: revenue cost per +100ms vs the measured lift. Inside the error budget and the lift wins -> promote, file the latency debt.
4. Otherwise buy the tail back: distill/quantize, cache features, or route the heavy model to high-value traffic only.

Roll back a model

Easy half: keep the previous version warm and flip the alias or traffic weight.

Hard half - a model leaves state behind:

• Predictions already written downstream stay wrong until you backfill.
• Items shown / transactions blocked are now in the training data, so the next retrain inherits the bad run.
• The old artifact may need a feature schema or library version that has since moved.

Make it rehearsed: pin each model version to its feature-pipeline version and image digest, keep last-known-good warm (not cold), tag every stored prediction with the model version that made it, and test the rollback before you need it.

Gate before production (roughly by hit rate)

1. Slice metrics vs the current production model, with a no-regression threshold on protected and high-value slices. Aggregate accuracy is the weakest gate and the one most teams stop at.
2. Serving parity: score a fixed sample through the real serving path, compare to offline. Any mismatch is skew.
3. Behavioral checks: known input-output invariants and directional expectations.
4. Load: p99 at expected peak, memory ceiling, cold-start time.
5. Schema contract: the model must reject unexpected types loudly, not silently impute.

Anything you cannot check before release becomes a canary metric.

Divide the latency budget (budget the tail, not the mean)

200ms p99 product budget -> ~120ms for the service after client network, gateway, rendering:

Per-hop timeouts must sum to less than the budget, not equal it. Fan-out breaks the arithmetic: 8 parallel feature calls cost the slowest of 8, so the group p99 is far worse than any single p99. Hedge, or accept a partial vector with defaults.

Cut inference latency, in this order

Measure the breakdown first - quantizing a model that costs 4ms of a 90ms request buys nothing.

1. Cut round trips: batch feature fetches into one call, cache slow-changing features near the server, precompute anything not request-dependent.
2. Shrink the model: distillation, quantization, pruning, smaller architecture, compiled runtime.
3. Serving mechanics: dynamic batching, warm pools, model pinned in memory.

Each lever has a cost: caching returns stale predictions, quantization loses a little quality, dynamic batching adds queue delay per request while raising throughput.

Reason about throughput vs latency under load

# Little's law: in_flight = arrival_rate * latency
in_flight = 2000 * 0.05   # -> 100 concurrent requests

• Waiting 10ms to fill a batch of 32 can triple GPU throughput and adds 10ms to every request. Larger batches hold more activation memory - the ceiling is usually memory, not compute.
• Past ~70% utilization, wait time grows faster than load and p99 becomes queue time, not model time.
• Adding replicas fixes tail latency far more reliably than shaving ms off the forward pass.
• Cap the batch wait window, not just batch size. Cap server concurrency so overload becomes a fast rejection, not a ten-second wait. Shed low-priority traffic before the tail collapses.

Set SLOs that mean something

• Derive from the caller: checkout has 300ms, model gets 80ms -> objective is p99 < 80ms.
• Measure at the client edge, rolling 28-day window. Server-side timers exclude queueing, feature fetch and network - they flatter you.
• Availability counts usable answers, not status codes. A timeout that returns a default is still an HTTP 200.
• Track p50 and p99 separately. Keep prediction quality as its own indicator - a service can be perfectly available and confidently wrong.
• Spend the error budget on purpose: comfortably inside -> ship riskier models faster; burned -> freeze releases.

Cut cost with traffic tiering

Rank requests by expected value, match each tier to a cheaper path: logged-in checkout -> full model; cache miss on a mid tier -> distilled/quantized model; crawlers, logged-out browsing, internal calls -> cached or precomputed. Precompute the few thousand hot entities in batch and reserve live inference for the long tail. Under pressure, degrade tiers in order rather than shedding traffic uniformly.

Tiering feeds itself: a segment routed to the weak model gets worse outcomes, which look like genuinely lower value next quarter.

• The notebook-to-package seam: a feature transform rewritten by hand diverges silently. Package the artifact with its inference code.
• Alerting on accuracy alone - labels arrive days late or never.
• Comparing a fresh challenger against a stale champion score.
• Pinning a query instead of pinning bytes.
• An image tag instead of an image digest.
• Untagged degraded responses feeding the next training set.
• A rollback path that has never been rehearsed.
• Mitigate first, diagnose second: flip to the last-good version, then find the renamed upstream column.