A p-value answers one narrow question: if the null hypothesis were true, how often would chance alone produce a result this extreme or more extreme? A small p-value means your data sits far out in the tail of what randomness usually produces.

It does not tell you the probability that the null is true. It does not tell you the probability your result is a fluke, and it says nothing about effect size. A p-value of 0.01 on a 0.05% conversion lift is statistically loud and commercially worthless.

It also does not survive abuse. Report a p-value from a test you stopped the moment it dipped under 0.05, and the number no longer means what the formula says. In practice I pair every p-value with the effect size and a confidence interval. The decision should rest on how big the change is, not on clearing a threshold.

Hypothesis testing is a structured way to decide whether an observed difference is bigger than routine noise. You assume nothing is happening, then measure how badly the data contradicts that assumption.

The walk-through I use:

• State the null and the alternative before looking at data. Null: the new checkout converts at the same rate as the old one.
• Pick the significance level and a test that matches the metric and the design.
• Compute the test statistic and its p-value from the collected sample.
• Compare against the threshold, then report the effect size and interval next to the verdict.

Order matters more than the arithmetic. Choosing the metric or the threshold after seeing results turns the procedure into storytelling. The most common production failure is not a wrong formula. It is a null hypothesis that got quietly rewritten once the numbers landed.

Type I error means you claimed an effect that is not there. Type II error means you missed an effect that is really there. The significance level sets your Type I rate, and power controls the Type II rate.

Costs differ by context, and that asymmetry should drive your threshold. Shipping a redesign that does nothing burns engineering time and pollutes future baselines. Killing a feature that actually worked costs revenue forever, quietly, because nobody measures the road not taken.

A medical screening test makes it concrete. A false positive sends a healthy person for an expensive biopsy. A false negative sends a sick person home untreated. With a fixed sample you cannot minimize both at once, since tightening one loosens the other. The only honest way to shrink both is to collect more data.

A 95% confidence interval comes from a procedure that captures the true value in 95% of repeated samples. For the interval you actually computed, the true value is either inside it or it is not. The 95% describes the method's long-run hit rate, not your one interval.

What it does not mean: there is no 95% probability that the parameter sits between your bounds. It is not a range containing 95% of users. It is also not a promise that 95% of future results will land inside it.

The practical read is the width. An interval of [-0.1%, +4.3%] on conversion says the change might be nothing, so you need more data before betting on it. A tight interval hugging zero is a genuinely different answer. It says the effect, if any exists, is too small to care about.

Correlation says two variables move together. Causation says changing one actually changes the other. Correlation is symmetric and cheap to compute, while causation is directional and needs either a mechanism or an intervention behind it.

Ice cream sales and drowning deaths rise together every summer. Nobody drowns because a stranger bought a cone. Hot weather drives both, so temperature is a confounder sitting behind the correlation.

A product version bites harder. Customers who use a company's mobile app spend more than web-only customers, so a team concludes the app drives spending. More likely the heaviest customers install the app first. Reverse causation and self-selection are both live explanations. The clean way to settle it is to randomize who gets nudged to install, then compare spend between the two randomized groups.

The central limit theorem says the sampling distribution of a mean approaches a normal shape as the sample grows, whatever shape the underlying data has. Average enough independent draws and the average behaves predictably, even when single values do not.

That is why it matters at work. Revenue per user is wildly skewed, with a long tail of big spenders and a pile of zeros. The mean of ten thousand users still behaves close to normal, so standard error formulas, z-tests, and confidence intervals stay usable on ugly data.

It does not fix everything. The heavier the tail, the larger the sample you need before the approximation is any good. When a handful of users can move the mean by themselves, trimming or capping extreme values works better than trusting the theorem and reporting a falsely tight interval.