Skip to content
Log in
← Statistics glossary

practical significance

Whether an effect is large enough to matter for the business, judged by effect size and cost, not by p-value.

Practical significance asks whether a difference is big enough to matter: worth the cost of building, shipping and maintaining the change. Statistical significance answers a different question: whether the difference is bigger than random noise.

The two come apart easily. With a huge sample, a lift of 0.1 percentage points can be highly significant and still earn less than the change costs. With a small sample, a lift that would be worth a lot can fail to reach significance simply because the test was too short.

To judge practical significance, look at the size of the effect and its confidence interval, and translate them into money or the metric the team cares about. Decide in advance what the smallest worthwhile effect is (the same number you use to plan the sample size). Then read the interval: if it lies entirely above that line, the change is both real and worth it; if it straddles the line, the data can't tell yet; if it lies below, the effect may be real but too small to care about.

The p-value alone can't tell you any of this. It says nothing about how big the difference is.

Example

CatChow tests a new product page on 400,000 visitors per variant. Conversion goes from 5.00% to 5.10%, p = 0.04: statistically significant.

The team had decided that the change is worth it only above +0.3 percentage points, because the new page doubles the photo budget. The 95% interval for the lift runs from about +0.004 to +0.2 points, entirely below +0.3. The effect is probably real, but it doesn't pay for itself, so the team keeps the old page.

Common mistakes

  • Celebrating a tiny significant lift. Check what it is worth before shipping.
  • Dismissing a large non-significant lift. It may be worth a longer test rather than a "no".
  • Defining "worth it" after the test. Set the smallest worthwhile effect before you see the results.
  • Reporting only the p-value. Always give the effect size and its interval so others can judge.

Learn it in the course