Statistics & Experimentation.
Cheat sheet.

• A p-value measures surprise under the null. The interval measures precision. Neither measures value.
• Decide on the effect size and its interval, not on clearing 0.05.
• Fix the metric, the sample size, and the end date before data lands. Everything chosen afterwards is storytelling.
• Randomization is the only assumption you control. Every other causal method trades it for a belief.

Say what a p-value does and does not mean

Read a confidence interval

A 95% CI comes from a procedure that captures the truth in 95% of repeated samples. Your one interval either contains it or not.

Not "95% probability the truth is in here", not "95% of users", not "95% of future results".

Convert significance level to confidence level

confidence = 1 - alpha. alpha = 0.05 pairs with a 95% interval, and for a standard two-sided test the interval excludes the null exactly when p < 0.05.

The equivalence breaks near the boundary for score/Wilson intervals, bootstrap intervals, and one-sided tests.

Choose one-tailed or two-tailed

Use one-tailed only when a move in the opposite direction leads to the same decision as no move. That is rare. Never pick the tail after seeing the data.

Name the error you are trading

With a fixed sample, tightening one loosens the other. Only more data shrinks both. Set alpha by which mistake costs more: shipping a dud burns time and pollutes baselines; killing a real winner costs revenue silently forever.

Quote a prediction interval for a single case

CI covers a parameter (the mean) and shrinks with n. A prediction interval covers the next single observation and never shrinks below the population spread.

mean delivery 3 days, 95% CI [2.9, 3.1]   // how well we know the average
                95% PI  [1, 7]            // what one parcel actually does

Quoting the CI to a customer is how you manufacture broken promises.

Compute sample size per arm

Inputs: baseline rate, minimum detectable effect (MDE), alpha, power.

// two-sided, alpha = 0.05, power = 0.80  ->  n ~= 16 * variance / effect^2
const variance = 0.05 * 0.95;            // 5% baseline conversion
const n = 16 * variance / (0.005 ** 2);  // 30,400 per arm for +0.5pp absolute

• Revenue-style metrics have far larger variance than rates - budget much more traffic.
• Each extra variant raises the count each arm needs.
• Sample size scales with 1 / effect^2: halving the MDE quadruples the traffic.

Pick the minimum detectable effect

The MDE comes from the decision, never from the traffic you have. Ask what lift would justify the build and the risk. If that number needs more users than a month supplies, say the test cannot answer the question.

Read 80% power correctly

Power is the chance of detecting an effect of the size you designed for. 80% power = a 1-in-5 miss rate at that size, and much worse for smaller true effects.

A flat result at 80% power is weak evidence of nothing. Report the interval, not a verdict.

Buy power with: more users, longer run, variance reduction (CUPED, stratification), or a bigger MDE.

Choose primary and guardrail metrics

Decide how long to run

• Minimum one full week, usually two: weekday and weekend users differ, and two weeks covers a purchase or renewal cycle.
• Hitting the sample size in two days does not end the test.
• Longer is not free: cookie churn dilutes assignment and the test queue backs up.
• Write the end date down before launch and let the analysis wait.

Bucket users deterministically

const bucket = hashInt(`${userId}:${experimentId}`) % 1000;
const arm = bucket < 500 ? 'control' : 'treatment';  // stable across sessions

Buckets are numbered slots: give 1% to a risky arm, widen to 50% later, reserve a range as a holdout, all without remapping anyone.

Two traps: no per-experiment salt means the same users are always in treatment; id % 2 skips the hash and maps signup order onto arms.

Pick the randomization unit

Analyze only users who were actually exposed to the surface under test; counting the unexposed shrinks the detectable effect.

Verify assignment was actually random

1. Sample ratio mismatch (SRM): chi-square the observed split against the intended one. 50.4/49.6 over 2M users is not luck - it is broken assignment or logging. Hard stop.
2. Pre-period balance: compare country, platform, tenure, and last month's primary metric across arms.
3. Do not over-read: check 20 covariates and one will hit p < 0.05 by chance. Covariate gaps need a much higher bar than SRM.

Usual causes: bots in one arm, a redirect dropping slow clients, exposure logged only inside the treatment code path.

The test came back flat

Read the interval first. [-3%, +4%] could never have seen a 2% lift - the honest word is inconclusive, not "no effect". Then check in order:

The in-flow lift is 40% but overall moved 2%

Dilution explains nearly every gap this size. 40% is on the 5% of traffic that reached the new flow.

• Check exposure share, whether the jump sits in one narrow segment, and whether 40% is relative on a tiny base (0.5% -> 0.7% is +40% relative, +0.2pp absolute).
• Report both: exposed-user effect says whether the feature works; all-assigned effect says what shipping is worth.

Quoting 40% to leadership when the company will only ever feel 2% is the trap.

A rate improved while volume fell

CTR rose because impressions (the denominator) fell. Look at the numerator alone: if clicks fell too, the product delivered less value.

• Stricter relevance filter = deliberate tradeoff, possibly fine. Rendering bug = not fine. Same rate move, opposite meaning.
• Decide on absolute clicks, sessions per user, or downstream conversion - far less gameable than a ratio.
• Ratio metrics need bootstrap over users or the delta method; treating impressions as independent units understates variance.

The winner faded after launch

Keep 5% of users on control after launch and read the gap over the following month.

Ten variants and one wins at p < 0.05

const anyFalsePositive = 1 - Math.pow(0.95, 10);  // 0.401 -> 40% chance of a fake winner

Someone peeked daily and stopped at 0.05

The 5% guarantee covers one planned analysis. Peeking daily for two weeks pushes the real false positive rate toward 20% or higher, and the effect at the stopping point is biased upward because you only stop when noise favors you.

If you must monitor: sequential / group sequential boundaries that spend the error budget across looks, or Bayesian monitoring. Fixed 0.05 plus daily peeking is the one option that quietly lies.

One metric rose and another dropped

Only ship if the exchange rate was agreed before the test ran.

• Guardrails (crash, latency, errors, unsubscribes) get a veto, not a trade.
• Genuine tradeoffs (revenue vs sessions) convert into one currency.
• Check the shape: a 1% average drop is often a 6% drop concentrated in new users.
• Check they are not one effect counted twice.
• Time asymmetry usually settles it: engagement gains fade, retention damage compounds. When close, ramp partially with a long-run holdback.

Decide not to run a test

Skip when the test cannot answer the question or costs more than the decision: low traffic (six months to power = stalling), legal/security/rebrand changes, strong network effects, one-shot launches, market-wide changes like pricing.

Test the assumption, not just the estimate: rerun the model on a fake launch date and confirm it finds nothing. Report the number with its assumption attached.

Limit interference between arms

Randomize above the boundary where users affect each other: cluster by market/city/graph community, switchback by time slice, or split the resource (inventory, driver supply, ad budget) per arm.

• Power now scales with the number of clusters, not users. Thirty cities detects only large effects.
• Cluster sizes are unequal: analyze cluster means or use ratio estimators with cluster-robust variance.
• Watch invisible shared state: caches, pacing budgets, and any model retrained on pooled logs carry treatment into control.

Fix standard errors when users are correlated

variance inflation ~= 1 + (m - 1) * ICC
ICC = 0.05, m = 100  ->  ~6x the reported variance

Use cluster means, cluster-robust standard errors, and a bootstrap that resamples clusters instead of users. Effective sample size is the number of independent units, not the number of events.

Separately, the estimand shifts: control users are partly treated by neighbors, so in a marketplace the measured lift can beat any achievable global gain. Model exposure (share of a user's neighbors in treatment), fit dose-response, and confirm with a market-level holdout.

Measure a metric that takes two months to mature

Split into a fast read and a slow read.

• Fast: a surrogate metric validated on past experiments where you hold both the early signal and the mature outcome. A surrogate that worked for pricing will mislead on onboarding.
• Slow: a long-term holdback - ship to most traffic, keep a small population untreated, read the mature metric when it lands. Start the clock early; extra traffic cannot buy calendar time.
• Holdbacks decay: two months of logged-out sessions, reinstalls, and device switches, and the attrition is not random. One permanent shared holdout beats a per-feature one.

Choose what to adjust for

Draw a DAG (nodes = variables, arrows = cause -> effect, no loops) before touching the data.

"Control for more variables" is not safer. And a DAG is an assumption: data cannot tell you which way an arrow points.

Spot Simpson's paradox

A trend holds in every subgroup and reverses when pooled, because the mix is unequal.

desktop:  B 10%  vs A 9%      B wins
mobile:   B  4%  vs A 3%      B wins
pooled:   B loses   (B is 80% mobile; mobile converts worse for everyone)

Split on a variable only when it causes the outcome and exists before assignment. Splitting on something the treatment itself caused hides the effect. Randomization usually balances the mix; quarter-over-quarter comparisons never do.

Separate correlation from causation

Correlation is symmetric and cheap; causation is directional and needs an intervention or a mechanism. When two variables move together, name the three rivals before believing the story: confounder, reverse causation, self-selection.

"App users spend more" is almost always heavy spenders installing first. Randomize who gets nudged to install.

Pick the distribution

• Binomial requires independent trials with the same p. Real users break both (repeat visits, heavy buyers), so true spread exceeds np(1-p) and your error bars are too narrow.
• Poisson assumes mean = variance. Incidents and hourly clustering break it - fitting Poisson to over-dispersed counts gives intervals that are far too tight.
• Normal sanity check with 68-95-99.7: mean 5 min, sd 4 puts two sd below zero, so the data is not normal. Skewed metrics like revenue break the rule badly.

Check the CLT actually applies

Needs: independent, identically distributed, finite variance.

• n = 30 is enough only for mildly skewed data. Revenue per user can stay skewed at n in the thousands - bootstrap the mean and look at its shape.
• Heavy tails (near-Cauchy: file sizes, insurance losses) never settle.
• Independence breaks first in practice: repeated events from one user mean effective n = number of users.
• When a handful of users can move the mean, cap or trim rather than trusting the theorem.

Build a confidence interval for a proportion

const r = k / n;                                  // 6/10 = 0.6
const half = 1.96 * Math.sqrt(r * (1 - r) / n);   // 0.304  (Wald)

At zero successes Wald collapses to zero width and claims certainty from nothing.

Get a two-tailed p-value from counts

// 1 head in 10 flips. H0: fair coin, p = 0.5, flips independent.
// p-value = P(X <= 1) + P(X >= 9) under Binomial(10, 0.5)
const p = 2 * (1 + 10) / 1024;  // 0.0215 -> reject fairness

The p-value is the tail, not the point probability of exactly what you saw. And 1 in 20 fair coins lands here - flip more.

Measure correlation

Plot the scatter first. Pearson 0.8 next to Spearman 0.3 means a handful of rows do all the work. Pearson near zero with an obvious U shape means you measured the wrong thing.

Pick a sampling design

Stratifying buys variance reduction only when the strata differ on the metric. With 3% enterprise customers, a random sample of 500 pulls ~8 of them and swings wildly.

Logs from a single region are a convenience sample in a warehouse costume.

Keep conditional probabilities pointing the right way

P(A and B) = P(A|B) * P(B). Sum a joint table across a row to get a marginal; divide a joint by a marginal to get a conditional.

1,000 users, 400 mobile, 100 of those churned
joint  P(mobile and churn) = 10%
marginal P(mobile)         = 40%
conditional P(churn|mobile)= 25%

"Most churners are on mobile" is P(mobile|churn). "Mobile users churn more" is P(churn|mobile). Only the second justifies rebuilding the app.

• Choosing the metric, the threshold, or the tail after seeing results.
• Stopping the moment p dips under 0.05.
• Filtering on post-assignment behavior ("engaged users only") - it rebuilds the bias randomization removed.
• Reporting a rate without its denominator, or a relative lift without the absolute.
• Calling an underpowered flat result "no effect".
• Adjusting for a mediator or conditioning on a collider.
• Trusting np(1-p) or Poisson variance on clustered real-user data.
• Averaging percentiles, or averaging a stratified sample without weights.
• Comparing a freshly retrained model against a stale control - that measures staleness, not the algorithm.
• Quoting a confidence interval as a promise about a single case.