Course recap: choosing the right test

You

Alex, I've learned twelve different methods over five modules and I can't remember which one goes where anymore.

Alex Tutor

That's exactly what today is for. Every method we covered answers one of a handful of questions. Once you know the question, the method mostly picks itself.

You

So it's not really twelve separate things to memorise?

Alex Tutor

Two things matter most: how many groups or variables you're comparing, and whether your data meets a parametric test's assumptions — roughly normal, no extreme outliers, decent sample size.

Alex Tutor

Question 1

CatChow compares time-on-page for 15 users in variant A vs 15 in variant B. The data are heavily skewed with a couple of extreme outliers. Which test fits best?

Choose one answerMann-Whitney U testIndependent-samples t-testPaired t-testOne-way ANOVA

Alex Tutor

For two independent groups: t-test if the data are roughly normal with a decent sample size, Mann-Whitney if they're skewed, small, or full of outliers.

You

And for the same users measured before and after a change?

Alex Tutor

That's paired data — paired t-test or its non-parametric twin, the Wilcoxon signed-rank test. Same normal-vs-skewed logic, just for differences instead of raw values.

You

Three or more groups — that's where ANOVA comes back?

Alex Tutor

Right: one-way ANOVA (parametric) or Kruskal-Wallis (non-parametric) for independent groups; repeated-measures ANOVA or Friedman for the same subjects measured repeatedly.

Alex Tutor

Question 2

CatChow measures the same 40 users' satisfaction score before and after a redesigned onboarding flow. Which test is appropriate?

Choose one answerPaired t-test (or Wilcoxon signed-rank if the differences are skewed)Independent-samples t-testPearson correlationk-means clustering

Alex Tutor

For a single A/B test on a conversion rate, you're comparing two proportions, not two means — that calls for a two-proportion z-test or a chi-square test.

You

And if I'm comparing conversion across 3 landing page variants at once?

Alex Tutor

Same family: a chi-square test on the 3-by-2 table of variant versus converted/not-converted, not ANOVA — ANOVA compares means of a continuous variable, and conversion is a proportion.

You

So 'is it a mean or a proportion' is another question I should be asking upfront?

Alex Tutor

Exactly. And whatever the test, wrap the result in a confidence interval, and plan the sample size before launch using baseline, MDE and power.

You

Give me a quick number to check I've got the CI formula right.

Alex Tutor

\text{margin of error} \approx 1.96 \times SE

The 95% margin of error: about 1.96 standard errors on each side of the estimate.

Here's a small one to compute by hand.

Alex Tutor

Question 5

CatChow's conversion rate estimate has a standard error of 1.2 percentage points. Using margin of error ≈ 1.96 × SE, what is the approximate 95% margin of error?

Your answer

Units: percentage points (e.g. 2.35)

Alex Tutor

When you want to know how two variables move together, that's correlation. When you want to predict one from another, that's regression.

You

And for a yes/no outcome, like 'will this user churn'?

Alex Tutor

Logistic regression: it predicts the probability of that yes/no outcome from your predictors.

You

Different from linear regression because the outcome is bounded between 0 and 1?

Alex Tutor

Exactly that. Try applying it to a concrete case.

Alex Tutor

Question 6

You want to predict the probability that a specific customer churns next month from their order history. Which method fits?

Choose one answerLogistic regressionLinear regressionOne-way ANOVAFactor analysis

When you have no target variable at all, just customers you want to group, that's this module: clustering.

You

And if it's survey questions I want to group instead of customers?

Alex Tutor

Factor analysis — same spirit as clustering, but grouping correlated variables into hidden themes instead of grouping observations into segments.

You

So clustering needs a decision about k, unless I use the hierarchical version?

Alex Tutor

Question 4

If you don't know in advance how many customer segments exist, hierarchical clustering is more appropriate than k-means.

TrueFalse

Alex Tutor

Let's put it all on one map. I'll give you several scenarios, and you match each to its method.

Alex Tutor

You

This is basically the cheat sheet I've wanted since module 2.

Alex Tutor

Keep it open, then — you'll want it for what's next.

You

Ready when you are.

Alex Tutor

Question 3

Match each situation to the method that fits it.

Tap an answer, then tap the row it belongs to. You can also drag.

Compare mean order value between 2 independent groups, roughly normal data, n = 60 each
Compare median session time between 2 groups with skewed data and outliers
Compare a metric across 4 marketing channels (roughly normal means)
Compare conversion rate (a proportion) across 3 landing page variants
Find hidden themes behind 20 correlated survey questions
Group customers into segments with no predefined labels

Answers left to place

Alex Tutor

One warning before we finish: picking the right test on paper doesn't protect you from the practical traps we covered in module 3.

You

Peeking at results early, running twenty metrics and reporting the one that's significant...

Alex Tutor

Exactly those. Decide the test and the primary metric before looking at data, and always check practical significance, not just p < 0.05.

You

So the process around the test matters as much as the test itself.

Alex Tutor

Question 7

Which of these are signs you should not trust a 'statistically significant' result at face value? Select all that apply.

Choose all that applyThe team checked results daily and stopped the day p dropped below 0.05The result also holds up on a separate holdout sampleIt's the one significant metric out of 20 tested, with no correction for multiple comparisonsThe effect size is tiny even though p < 0.05

Let's cement the whole process — not just which test, but the order you do things in.

You

I have a feeling 'pick the test first, look at data second' is about to become a strict rule.

Alex Tutor

It already is one. Order these steps and you'll never method-shop again.

You

Okay, let me think through the sequence.

Alex Tutor

Question 8

Put the steps of a disciplined test-selection process in the right order.

Tap an item, then tap where it should go. You can also drag or use the arrows.

Alex Tutor

That's the whole course in one map: how many groups, what shape the data has, and what question you're really asking.

You

Six months ago I would've just eyeballed two averages and called it a day.

Alex Tutor

And now you know why that's not enough.

You

So what's left before I graduate this course?

Alex Tutor

Explain your test-selection process to a colleague — that's the next task.

Alex Tutor

Question 9Short answer · AI-checked task

In 2–3 sentences, explain how you'd decide between a t-test and a Mann-Whitney test for comparing average order value between two CatChow markets.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Last one: leadership wants a single, durable guide the whole company can use, not another one-off analysis.

You

No pressure, just writing the company's statistics constitution.

Alex Tutor

You have everything you need for it already — every method, every pitfall, all six modules.

You

Alright. Let's finish this course properly.

Alex Tutor

Question 10Case · AI-checked task

Write your method-selection memo below.

CatChow's leadership asks you to draft a short internal guide so every team picks statistical methods consistently instead of each analyst improvising. The guide should help someone decide what to use and warn them away from common mistakes, without requiring them to have taken this whole course.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

That's a wrap. You started this course eyeballing averages, and now you can defend a test-selection decision in front of anyone. Well done, Sofia.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Back to the course