statistical significance

A result is significant when p < alpha; it says the effect is unlikely to be pure noise, not that it is large or important.

A result is statistically significant when its p-value falls below the significance level α you fixed before the test, usually 0.05. It means the observed difference would be unusual if there were no real effect, so "it's just noise" becomes an unconvincing explanation. That is all it means. It says nothing about whether the effect is large, valuable or worth shipping.

Practical significance asks the business question: is the effect big enough to matter once you account for costs, risks and effort? You answer it with an effect size in meaningful units (absolute lift in percentage points, revenue per user, minutes saved) and its confidence interval, compared against a threshold agreed before the test. That threshold is often the minimum detectable effect (MDE) you used to plan the sample size.

The two can disagree in both directions. With huge samples the standard error becomes tiny, so commercially trivial differences turn statistically significant. With small samples a genuinely useful effect can stay non-significant because the test lacks statistical power.

A practical way to read any result is to look at the confidence interval for the effect:

the whole interval is above your practical threshold: significant and worth acting on;
it excludes zero but sits below the threshold: real, but too small to matter;
it includes zero and also values above the threshold: inconclusive, so collect more data.

Remember too that α = 0.05 means accepting a 5% false-positive rate per test. Run twenty tests on changes that do nothing and you should expect about one "significant" winner.

Example

CatChow shows a free-delivery banner to half of 1.2 million visitors.

	Visitors	Orders	Conversion
Control	600,000	24,000	4.00%
Banner	600,000	24,600	4.10%

Statistical significance. Pooled rate $\bar p = 48{,}600 / 1{,}200{,}000 = 0.0405$ . $SE = \sqrt{0.0405 \times 0.9595 \times \frac{2}{600{,}000}} \approx 0.00036$ , so $z = 0.0010 / 0.00036 \approx 2.78$ and $p \approx 0.005$ . Clearly significant.

Practical significance. The lift is +0.10 percentage points (2.5% relative). The 95% confidence interval for the difference uses the unpooled standard error, which here is also about 0.00036: $0.10 \pm 1.96 \times 0.036 \approx 0.10 \pm 0.07$ , i.e. +0.03 to +0.17 pp.

Free delivery is expensive. Finance calculated that the banner pays for itself only if it lifts conversion by at least +0.3 pp. The entire interval sits below that line.

Verdict: the effect is real (statistically significant) but too small to cover its cost (not practically significant). Don't ship it as is, but it's worth trying a cheaper version, such as free delivery above a minimum basket.

Common mistakes

Hearing "significant" as "important". In a slide deck, write "statistically significant" and put the size of the effect right next to it.
Shipping without a practical threshold. If nobody agreed in advance what lift is worth the cost, any significant result will look like a win.
Calling a non-significant result "no effect". If the confidence interval still includes lifts you would care about, the honest answer is "we don't know yet".
Hunting for significance. Slicing results by segment, metric and day until something crosses 0.05 guarantees false winners; correct for multiple comparisons or pre-register what you'll test.
Comparing verdicts instead of effects. "Test A was significant and test B wasn't, so A is better" is wrong: the two effects may be nearly identical. Compare the effects and their intervals directly.

Learn it in the course

p-value and significance · What a p-value really says, what it doesn't, and statistical versus practical significance.