null hypothesis (H0)

The default claim of no difference or no effect, e.g. 'the new checkout does not change conversion'.

Every statistical test starts with two competing claims. The null hypothesis $H_0$ is the boring default: there is no difference, no effect, nothing going on. The alternative hypothesis $H_1$ is the claim you would like to support: there is a difference. In an A/B test on conversion:

$H_0: p_B = p_A \qquad H_1: p_B \neq p_A$

Why start from "nothing happened"? Because it is the only claim precise enough to calculate with. Assuming $H_0$ is true, we can work out how much the results would wobble from sampling noise alone, and then ask how surprising our actual data would be in that world. That surprise is the p-value. If $p < \alpha$ (usually 0.05), the data are hard to square with $H_0$ , and we reject it in favour of $H_1$ . If not, we fail to reject $H_0$ .

The wording matters. Failing to reject $H_0$ does not prove there is no effect; it means this test, with this sample, did not find convincing evidence. Think of a court: "not guilty" is not the same as "innocent". Likewise, rejecting $H_0$ says an effect is probably not zero, not that it is large or worth shipping; that is a question of effect size and practical significance.

The alternative can be two-sided ("B differs from A", the safe default) or one-sided ("B is better than A"). Choose it, along with $\alpha$ and the primary metric, before you look at the results. Picking the direction after seeing the data quietly doubles your chance of a false positive, the type I error.

Example

CatChow tests a checkout redesign: 1,000 users see the old checkout (A), 1,000 see the new one (B).

Hypotheses, written before launch:

$H_0$ : the redesign doesn't change conversion, $p_B = p_A$
$H_1$ : it does change conversion, $p_B \neq p_A$ (two-sided, $\alpha = 0.05$ )

Results: A converts 52 of 1,000 (5.2%), B converts 68 of 1,000 (6.8%). The difference is 1.6 percentage points.

Test under $H_0$ . If there were truly no difference, both groups would share one conversion rate, estimated by pooling: $120 / 2{,}000 = 0.06$ . The standard error of the difference is

$SE = \sqrt{0.06 \times 0.94 \times \left(\tfrac{1}{1000} + \tfrac{1}{1000}\right)} = \sqrt{0.0001128} \approx 0.0106$

so $z = 0.016 / 0.0106 \approx 1.51$ . The two-tailed p-value is about 0.13.

Decision. $0.13 > 0.05$ , so we fail to reject $H_0$ . The correct summary is not "the redesign doesn't work" but "a 1.6 pp gap this size shows up about 13% of the time from noise alone, so 1,000 users per group can't tell us yet". The 95% confidence interval for the difference, about −0.5 to +3.7 pp, still includes zero and also includes a meaningful win. The next step is a larger test, not a verdict.

Common mistakes

Saying "we accepted H0" or "the test proved there's no effect". A non-significant result means the evidence was not strong enough, often because the sample was too small.
Writing the hypotheses after seeing the data. Choosing one-sided H1 or a new metric once you know which way the numbers went inflates false positives.
Treating "reject H0" as "H1 is big and important". With huge samples, trivial differences become significant; check the effect size and confidence interval.
Stating H0 about the sample instead of the population. The sample means are what they are; hypotheses are about the true conversion rates or averages behind them.
Testing a vague H1. "The redesign improves things" needs a metric, a direction and, ideally, a minimum effect worth detecting before the test starts.

Learn it in the course

p-value and significance · What a p-value really says, what it doesn't, and statistical versus practical significance.
The t-test · Comparing the means of two groups and deciding whether the gap is bigger than chance.
Power and sample size · How many users an experiment needs to detect the effect you care about.
ANOVA: comparing many groups · Testing three or more variants at once without inflating false positives.