Power and sample size

You

Let's launch the checkout redesign test to 200 users each and check back Friday.

Alex Tutor

Before we do - what effect are we trying to catch, and can 200 users even show it if it's real?

You

I... hadn't thought about that part.

Alex Tutor

That's statistical power: the probability your test detects a real effect, if one truly exists. Low power means real wins get missed.

Alex Tutor

Miss a real effect and you either ship nothing when you should have, or worse - you decide the idea 'doesn't work' and drop it for good.

Alex Tutor

Question 1

Higher statistical power means a lower chance of a type II error (failing to detect a real effect that exists).

TrueFalse

Four things drive power: your sample size, the effect size you care about, your significance level (α), and the data's natural variability.

You

Effect size... you mean like a percentage lift?

Alex Tutor

In conversion tests, we usually call it the minimum detectable effect (MDE) - the smallest true difference you actually want your test to be able to catch.

Smaller MDE = harder to catch = you need way more users. It's the single biggest lever on required sample size.

(Side note: unlike Module 1's AOV example, this lesson works with proportions, not an arithmetic average - the formulas differ slightly.)

Let's size a real one. CatChow's checkout page converts at a baseline p1 = 5.0%. Marketing wants to detect a rise to p2 = 6.0% - that's the MDE.

Alex Tutor

n \approx \frac{(z_{\alpha/2}+z_{\beta})^2 \left[p_1(1-p_1)+p_2(1-p_2)\right]}{(p_1-p_2)^2}

Approximate sample size per variant for a two-proportion test at significance level α (two-sided) and power 1 - β.

For the usual α = 0.05 (two-sided) and 80% power, the critical values are z = 1.96 and z_β = 0.84.

Alex Tutor

Question 2

Compute p1(1-p1) + p2(1-p2) for p1 = 5.0% and p2 = 6.0% (as a decimal proportion, not a percentage).

Your answer

Units: (e.g. 0.0500)

Plugging in: n ≈ (1.96+0.84)² × 0.1039 / (0.05-0.06)² = 7.84 × 0.1039 / 0.0001 ≈ 8,146 users per variant.

You

8,146 per variant? That's over 16,000 total for a 1-point lift?!

Alex Tutor

Yes - small baseline rates and small absolute lifts are expensive to detect. That's the real cost of chasing tiny effects.

And it's sharply non-linear: shrink the MDE and the required sample explodes, because it's the squared difference in the denominator.

Alex Tutor

Question 3

If you halve the MDE you're targeting (keeping α, power and baseline variability fixed), the required sample size is multiplied by about how much?

Your answer

Units: x (e.g. 2)

So what do you do when the sample-size calculator says 8,146 but your test page only gets 3,000 visits total in two weeks?

You

Just run it for two weeks and check whatever p-value comes out?

Alex Tutor

That test would be badly underpowered - likely to miss a real 1-point lift even if it's there. You have real options instead.

Accept a bigger MDE (only detect a 2-point jump, not 1), run longer to accumulate the traffic, or test on a higher-traffic page instead.

Alex Tutor

Question 4

The calculator says you need 8,146 users per variant; you only get 1,500/week. What's the honest move?

Choose one answerRun for 2 weeks regardless and report whatever p-value you getLower α to 0.10 so the calculator needs fewer usersAccept a larger MDE and/or extend the test until you reach the required sampleAdd five more metrics so something is bound to be significant

Alex Tutor

Let's pin down the vocabulary - power ties directly to the two error types from the significance-testing lessons.

You

Type I, type II... I always mix those up.

Alex Tutor

Fair - most people do. Match each term below and it should click.

You

Okay, throw it at me.

Alex Tutor

Question 5

Match each term to its definition.

Tap an answer, then tap the row it belongs to. You can also drag.

Type I error (α)
Type II error (β)
Statistical power (1 - β)
Minimum detectable effect (MDE)

Answers left to place

Nailed it. Now let's check which levers actually change how many users you need.

You

I'll guess: smaller MDE needs more people.

Alex Tutor

Right instinct. There are three more levers just like that one.

You

Bring on the multi-select.

Alex Tutor

Question 6

Which of these increase the required sample size, holding everything else fixed? Select all that apply.

Choose all that applyChoosing a smaller minimum detectable effectRequiring 95% power instead of 80%Using a stricter significance level (α = 0.01 instead of 0.05)Choosing a larger minimum detectable effect

Alex Tutor

One more habit to build: deciding sample size before the test starts, not after seeing early results.

You

Actually... our team's rule is every test runs exactly one week, no exceptions. 'That's our process.'

Alex Tutor

That's a trap disguised as consistency. A week's worth of traffic isn't the same as a week's worth of power.

You

Because traffic isn't constant...

Alex Tutor

Question 7Short answer · AI-checked task

In 2-3 sentences, explain to Sofia's teammate why fixing the test duration at 'always one week' is worse than fixing the required sample size.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Good - now a messier, real situation to reason through.

You

Bring it.

Alex Tutor

CatChow's checkout page gets about 500 sessions/week. Marketing wants to test a new 'subscribe & save' banner, targeting a lift from 5.0% to 6.0% conversion.

You

That's the exact example from before - 8,146 per variant, right?

Alex Tutor

Question 8Case · AI-checked task

Should CatChow launch this test on schedule?

CatChow's checkout page gets ~500 sessions/week (250 per variant if split evenly). Marketing wants to detect a lift from a 5.0% baseline to 6.0% (the same MDE as our worked example, needing approximately 8,146 users per variant at 80% power and α = 0.05). At 250 users/variant/week, reaching that sample takes roughly 33 weeks. Marketing wants results in a month. Write a one-paragraph recommendation.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

That's the real skill: catching an underpowered plan before it wastes a month proving nothing.

You

Next: actually running the conversion-rate test properly, right?

Alex Tutor

Exactly - proportions, the z-test, and reading uplift without fooling yourself.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Conversion rates and proportion tests