ANOVA: comparing many groups

Alex Tutor

Sofia, remember our three onboarding email drafts? Marketing finally ran the test: new users got the Classic, Playful or Discount-first version.

You

And Discount-first clearly wins, right? People love discounts.

Alex Tutor

Maybe. We measured first-month spend for 5 new users per variant. Before comparing means, we need a test built for three groups at once, not two.

You

Why not just run a t-test for each pair: Classic vs Playful, Playful vs Discount, Classic vs Discount?

Alex Tutor

Each t-test risks a 5% false alarm on its own. Run three, and the chance that at least one screams "significant!" by pure luck climbs well above 5%.

Alex Tutor

Question 1

CatChow tested three onboarding emails on 5 new users each and wants to compare average first-month spend across all three. Why is running three separate t-tests (one per pair) a bad idea?

Choose one answerIt takes too long to compute three tests instead of oneEach extra test adds its own chance of a false positive, so the overall error rate climbs above 5%t-tests only work with exactly two groups, so the spreadsheet will refuse to run themThree t-tests would always agree with each other, so it's redundant but never wrong

Alex Tutor

Analysis of variance (ANOVA) runs one test for all three groups. It asks: is the spread between group means big compared with the spread within each group?

Alex Tutor

F = \dfrac{MS_{between}}{MS_{within}}

The F-statistic: between-group mean square divided by within-group mean square.

Between-group variance measures how far the three variant means sit from the overall average. Big differences between variants push it up.

Within-group variance is the noise: how much users inside the same variant differ from their own group's mean. You'd see it even with no real effect at all.

You

So a big F just means the variants are different somehow?

Alex Tutor

Right idea. ANOVA's null hypothesis (H0) says all three population means are equal. A large F is evidence against that, but it doesn't yet say which variant is different.

Alex Tutor

Question 2

The null hypothesis of a one-way ANOVA is that at least one group mean differs from the others.

TrueFalse

Alex Tutor

Here's first-month spend, in USD, for the 5 users in each variant: Classic 18, 19, 20, 21, 22; Playful 23, 24, 25, 26, 27; Discount-first 28, 29, 30, 31, 32.

Alex Tutor

Average first-month spend by variantThe three variant means are evenly spaced 5 USD apart, while users inside each variant vary by only a couple of dollars.

Chart data

Email variant	Mean spend
Classic	20
Playful	25
Discount-first	30

Grand mean across all 15 users: (20 + 25 + 30) / 3 = 25 dollars. Each variant's mean sits exactly 5 dollars from it.

Alex Tutor

SS_{between} = n\sum_{j=1}^{k}(\bar{x}_j - \bar{x})^2

Between-group sum of squares: n users per group times the squared distance of each group mean from the grand mean.

With n = 5 per group: SS_between = 5 × ((20−25)² + (25−25)² + (30−25)²) = 5 × (25 + 0 + 25) = 250.

Within-group SS adds each user's squared distance from their own group mean: 10 + 10 + 10 = 30. Three groups leave 15 − 3 = 12 degrees of freedom for it.

Alex Tutor

Question 3

Using SS_between = 250 (df = 2) and SS_within = 30 (df = 12) from the CatChow email test, compute the F-statistic.

Your answer

Units: F-value, e.g. 12.5

F = 50 is enormous. For df(2, 12), a result this extreme would happen by chance far less than 1 in 1000 times if the variants truly performed the same.

You

So it's decided: Discount-first is the winner, we ship it everywhere?

Alex Tutor

Not yet. ANOVA only tells us the three means aren't all equal. It doesn't say whether Discount-first beats Playful, or just barely beats Classic.

Comparing all pairs safely needs a Bonferroni correction: divide alpha by the number of comparisons to keep the overall risk near 5%.

Alex Tutor

Question 4

You plan to compare all 3 pairs of variants (Classic–Playful, Playful–Discount, Classic–Discount) after the significant ANOVA, using a Bonferroni correction to keep the family-wise error rate at 0.05. What significance threshold (alpha) should each pairwise test use?

Your answer

Units: alpha, e.g. 0.01

Alex Tutor

This safe pairwise comparison after a significant ANOVA is called a post hoc test. The common choice for equal-sized groups is Tukey's HSD test.

You

HSD... sounds dramatic. What does it actually do?

Alex Tutor

It compares every pair of group means using one shared threshold, built from the within-group variance, so the overall false-positive rate across all pairs stays near 5%.

Run it here and Discount-first beats both Classic and Playful, and Playful also beats Classic. All three differ, matching how far apart their means already looked.

Alex Tutor

Question 5

Match each ANOVA term to what it describes.

Tap an answer, then tap the row it belongs to. You can also drag.

Between-group variance
Within-group variance
F-statistic
Post hoc test

Answers left to place

Alex Tutor

ANOVA assumes roughly normal data and similar variances in each group, the same assumptions as the t-test.

When those are badly broken (tiny samples, wild outliers), use the Kruskal-Wallis H test instead: Mann-Whitney's idea, extended to three or more groups.

You

Ranking again, like we did for two groups?

Alex Tutor

Exactly the same spirit: convert every value across all groups to its rank, then compare rank sums instead of means. It trades a little power for safety against outliers.

Alex Tutor

Question 6

Which statements about one-way ANOVA and its F-statistic are true? Select all that apply.

Choose all that applyA large F means the between-group variance is large relative to the within-group varianceANOVA tells you exactly which pair of groups differsANOVA assumes roughly normal data with similar variances across groupsWith very skewed data or tiny samples, the Kruskal-Wallis H test is a safer alternativeRunning one ANOVA instead of three t-tests makes the overall false-positive rate higher

Alex Tutor

One ANOVA beats three t-tests: one honest false-positive rate, one F-statistic, and within-group noise as the baseline to compare against.

You

So next time marketing says "the third variant obviously won", I get to ask "obviously, or F-statistically"?

Alex Tutor

Almost. Also look at the actual group means and the chart, not just F. A huge F with a tiny, meaningless dollar difference isn't a business win.

Two tasks left: explain the post hoc step in your own words, then advise on a slightly trickier CatChow test.

Alex Tutor

Question 7Short answer · AI-checked task

In 2–4 sentences, explain why a significant one-way ANOVA result on its own doesn't tell CatChow which onboarding email variant to ship.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Nice reasoning. Let's stress-test it with a trickier, real CatChow case.

This one won't hand you a tidy F = 50; the result sits right on the fence.

You

Great, so I actually have to think instead of just reading the F-statistic?

Alex Tutor

Exactly. That's the whole point of statistics literacy for PMs.

Alex Tutor

Question 8Case · AI-checked task

What would you tell marketing, and what would you do next?

CatChow also tested four homepage banner designs (A–D) on 6 new users each, measuring day-7 retention. The one-way ANOVA gives F = 2.95 for df(3, 20); the critical F at alpha = 0.05 is about 3.10. Marketing wants to declare Banner D the winner immediately, because its raw mean retention is the highest of the four.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Multifactor ANOVA and interactions