A/B test sample size: how many users you need

Your A/B test sample size depends on four inputs: the baseline rate of your metric, the minimum detectable effect (MDE) you care about, the significance level (usually 5%) and the statistical power (usually 80%). Plug them into a standard formula or calculator and you get the number of users per variant; divide the total by your daily traffic and you get the test duration. For a 4% conversion rate and a 10% relative lift, that works out to roughly 39,500 users per group. The rest of this article shows where that number comes from, how to shrink it honestly, and which shortcuts quietly break your results.

Why "let's run it for a week and see" fails

Picture a Monday stand-up at CatChow, an online pet-food shop. The team has redesigned the checkout page, and someone suggests: "Let's show it to half the visitors for a week and look at the numbers on Friday."

It sounds reasonable, and it is how many teams burn a sprint. A week of traffic might be enough to see a huge effect, but checkout changes rarely produce huge effects. If the redesign lifts conversion by a modest 5–10%, a one-week test will most likely come back "not significant". The team then concludes the redesign "doesn't work" and moves on, when in fact the test simply never had the resolution to see the effect.

Sample size planning answers one question before launch: how much data do we need so that a real effect of the size we care about has a good chance of showing up?

The four inputs

1. Baseline rate

The current value of the metric in the control group. For CatChow's checkout, 4 out of every 100 visitors place an order, so the baseline conversion rate is 4%. Take it from recent data (the last 4–8 weeks), not from a good month you remember fondly.

The baseline matters because rare events are harder to measure. Detecting a change in a 2% conversion rate takes far more users than detecting the same relative change in a 10% rate.

2. Minimum detectable effect (MDE)

The smallest effect that would be worth acting on. It is a business decision, not a statistical one. Ask: "If the redesign lifts conversion by X, would we ship it and would it pay for itself?"

State the MDE clearly as relative or absolute. A 10% relative lift on a 4% baseline means going from 4.0% to 4.4%, which is an absolute change of 0.4 percentage points. Mixing these two up is the most common source of wildly wrong sample sizes.

3. Significance level ($\alpha$)

The false-positive rate you accept: the chance of declaring a winner when there is no real difference. The convention is $\alpha = 0.05$, tested two-sided.

4. Statistical power ($1 - \beta$)

The probability that the test detects the effect if it really is as large as your MDE. The convention is 80%, meaning you accept a 20% chance of missing a real effect of that size. For decisions that are expensive to get wrong, teams use 90%.

The formula for conversion rates

For comparing two proportions, the number of users you need in each group is approximately:

$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot \big(p_1(1-p_1) + p_2(1-p_2)\big)}{(p_2 - p_1)^2}$

Here $p_1$ is the baseline rate, $p_2$ is the rate you hope to see in the variant, and the $z$ values come from the normal distribution: $z_{0.975} \approx 1.96$ for a two-sided 5% test and $z_{0.80} \approx 0.84$ for 80% power.

You don't need to memorise it. Any decent online calculator or a line of Python does this. But it pays to understand what is in it: the difference $p_2 - p_1$ sits in the denominator squared. That single fact explains most of the surprises below.

Worked example: CatChow's checkout redesign

Inputs:

• baseline conversion $p_1 = 0.04$;
• MDE: 10% relative lift, so $p_2 = 0.044$;
• $\alpha = 0.05$ two-sided, power 80%.

Step by step:

1. $(1.96 + 0.84)^2 \approx 7.85$.
2. Variance terms: $0.04 \times 0.96 = 0.0384$ and $0.044 \times 0.956 = 0.0421$; together $0.0805$.
3. Squared difference: $(0.044 - 0.04)^2 = 0.004^2 = 0.000016$.
4. $n \approx 7.85 \times 0.0805 / 0.000016 \approx 39{,}500$.

The exact figure is 39,473 users per group, or about 79,000 in total.

CatChow's checkout sees about 3,000 visitors a day, split 50/50. So the test needs $78{,}946 / 3{,}000 \approx 26.3$ days. Round up to whole weeks: run it for 4 weeks (28 days), so that every weekday is represented equally. Shopping on a Sunday evening looks nothing like shopping on a Tuesday morning.

A quick sanity check: a handy rule of thumb for 80% power and $\alpha = 0.05$ is $n \approx 16 \cdot p(1-p) / \delta^2$, where $\delta$ is the absolute difference. Here that gives $16 \times 0.0384 / 0.000016 = 38{,}400$, close to the exact result. If your calculator gives a number ten times bigger or smaller, you probably mixed up relative and absolute MDE.

How the inputs move the number

Keeping the 4% baseline, $\alpha = 0.05$ and 80% power, and changing only the MDE:

Halving the MDE roughly quadruples the sample. That is the squared difference at work.

Other levers, all for the 10% relative MDE:

• Power 90% instead of 80%: 52,842 per group (about a third more).
• $\alpha = 0.01$ instead of 0.05: 58,734 per group.
• Baseline 2% instead of 4%: 80,679 per group. Baseline 10%: 14,749 per group.

What about average order value and other continuous metrics?

For a metric like average order value (AOV) the logic is the same, but the variance comes from the data's standard deviation $\sigma$ instead of $p(1-p)$:

$n = \frac{2 \, (z_{1-\alpha/2} + z_{1-\beta})^2 \, \sigma^2}{\delta^2}$

If CatChow's AOV is about 40 dollars with a standard deviation of 25 dollars and you want to detect a 2-dollar change, then $n = 2 \times 7.85 \times 625 / 4 \approx 2{,}453$ users per group. Be careful: revenue data is usually skewed, with a few very large orders inflating $\sigma$. Estimate the standard deviation from real recent orders, and consider capping extreme values before you plan.

Common mistakes

Picking the MDE to fit the traffic. Working backwards ("we have two weeks, so the MDE is whatever two weeks can detect") is fine as long as you say it out loud: "This test can only catch a lift of 20% or more." If nobody believes the change could do that, don't run the test, or pick a more sensitive metric.

Peeking and stopping early. Checking the dashboard every day and stopping the moment $p < 0.05$ inflates the false-positive rate far above 5%. The sample size is a commitment: decide it up front and analyse once, or use a method designed for continuous monitoring (sequential testing).

Stopping mid-week. Even if you hit the target on day 23, finish the full week. Otherwise the result over-represents some days of the week.

Randomising by one unit and counting another. If you randomise users but count sessions, the observations aren't independent and the real sample is smaller than it looks. Plan the sample in the same unit you randomise.

Forgetting about multiple variants and metrics. Three variants against control, or ten "primary" metrics, mean more chances for a false positive. Either adjust $\alpha$ (for example, a Bonferroni correction) and plan for a larger sample, or pick one primary metric in advance.

Ignoring a sample ratio mismatch. If you planned a 50/50 split and got 52/48 on large numbers, something in the assignment is broken. Fix that before trusting any result.

Key takeaways

• A/B test sample size is set by four inputs: baseline, MDE, $\alpha$ and power.
• The MDE is a business choice: the smallest lift worth shipping.
• Sample size grows with the square of the precision you want: halve the MDE, and you need about four times the users.
• Convert users to days with your real traffic and round up to whole weeks.
• Decide the sample size before launch, and don't stop early just because the p-value dipped below 0.05.

FAQ

What is a good sample size for an A/B test?

There is no universal number. It depends on your baseline rate and the smallest effect you care about. With a 4% conversion rate, detecting a 10% relative lift at 80% power needs about 39,500 users per group, while detecting a 30% lift needs under 5,000. Calculate it for your own inputs.

How long should an A/B test run?

Divide the total required sample by the daily traffic entering the test, then round up to full weeks. Most product tests should run at least one full week, and two is a safer minimum when behaviour differs between weekdays and weekends.

What if I don't have enough traffic?

Test bolder changes (a larger MDE), use a metric higher up the funnel with a higher baseline (for example, add-to-cart instead of purchase), reduce variance by capping outliers, or accept that this decision will be made on judgement rather than a test. Running an underpowered test and reading its result as "no effect" is the worst option.

Should I use 80% or 90% power?

80% is the common default. Use 90% when missing a real effect would be costly, for example when a "no effect" result would kill a strategic project. Expect about a third more users for the same MDE.

Learn it hands-on

Sample size planning clicks once you have done it a few times with real numbers. The free lesson Power and sample size walks through CatChow's checkout test step by step, with quizzes and a practice task checked by AI. It is part of the free course Statistics for Product Managers and Marketers.