p-value

The probability of getting a result at least as extreme as the observed one if H0 were true. It is not the probability that H0 is true.

A p-value answers one narrow question: if there were truly no difference (the null hypothesis, H0, is true), how often would random sampling alone produce a result at least as extreme as the one you observed? A small p-value means your data would be surprising in a world without an effect, which gives you grounds to reject H0. A large p-value means the data sit comfortably with pure noise.

It comes from a test statistic. First you squeeze the data into one number that measures the gap relative to its noise, for example $z = \frac{\text{difference}}{\text{standard error}}$ or Student's $t$ . Then you ask how much of that statistic's distribution under H0 lies at or beyond the value you got. For a two-tailed test that is $p = P(|Z| \ge |z_{\text{obs}}|)$ , assuming H0.

The decision rule is short: pick a significance level α before the test (usually 0.05). If $p < \alpha$ , the result is statistically significant and you reject H0; otherwise you don't.

Just as important is what the p-value is not. It is not the probability that H0 is true, not the probability that your variant really wins, and not a measure of how big the effect is. Because the standard error shrinks as the sample grows, the same small gap produces an ever smaller p-value with more users. That's why a p-value should always travel with an effect size and a confidence interval: they say how large the difference is and how precisely you have measured it.

Example

CatChow tested a redesigned checkout on 1,000 users per variant. Control: 52 orders (5.2%). Treatment: 68 orders (6.8%).

Pooled conversion rate: $\bar p = \frac{52 + 68}{2000} = 0.06$ .
Standard error under H0: $SE = \sqrt{0.06 \times 0.94 \times \left(\frac{1}{1000} + \frac{1}{1000}\right)} = \sqrt{0.0001128} \approx 0.0106$ .
Test statistic: $z = \frac{0.068 - 0.052}{0.0106} \approx 1.51$ .
Two-tailed p-value: $P(|Z| \ge 1.51) \approx 0.13$ .

How to read it: if the redesign changed nothing, a gap of 1.6 percentage points or more would still show up in about 13% of tests like this one, purely by chance. That is above α = 0.05, so we don't reject H0.

How not to read it: "there's a 13% chance the redesign is useless" or "the redesign has no effect". The test was simply too small to tell.

Now keep the same rates but run 4,000 users per variant (208 vs 272 orders). The pooled rate is still 0.06, but $SE = \sqrt{0.0564 \times 0.0005} \approx 0.0053$ , so $z \approx 0.016 / 0.0053 \approx 3.01$ and $p \approx 0.003$ . Same effect, very different p-value: only the sample size changed.

Common mistakes

Reading p as the chance that H0 is true. p = 0.03 does not mean "3% chance there's no effect" or "97% chance the variant wins". It is a probability about the data, computed assuming H0.
Treating a tiny p-value as a big effect. With hundreds of thousands of users, a 0.05-point lift can give p < 0.001. Check the lift and its confidence interval before celebrating.
Treating p > 0.05 as proof of no effect. An underpowered test fails to reject H0 even when the change works. Say "inconclusive", not "it doesn't work".
Peeking and fishing. Checking the dashboard daily and stopping the moment p dips below 0.05, or testing ten metrics and reporting the one that "worked", pushes the real false-positive rate far above 5%.
Treating 0.05 as a cliff. p = 0.049 and p = 0.051 carry almost identical evidence. Fix α before the test and report the actual p-value, not just "significant / not significant".

Learn it in the course

p-value and significance · What a p-value really says, what it doesn't, and statistical versus practical significance.
Power and sample size · How many users an experiment needs to detect the effect you care about.