The t-test

Alex Tutor

Sofia, the checkout team added an upsell widget ('add a scratching post?') to half of new sessions last week. They want to know: does it change the average order value?

You

Easy — control averaged

42.50, variant averaged

52.50. Widget wins, ship it everywhere!

Alex Tutor

Whoa, hold the confetti. Both numbers came from samples of only 10 orders each. Remember last lesson: sample means wobble. We need a way to ask 'is $10 bigger than the wobble, or is it the wobble?'

You

So we can't just trust whichever number is bigger?

Alex Tutor

Not on its own. That's exactly what the t-test for independent samples answers: is the gap between two group means bigger than sampling noise would plausibly produce? Here's the full dataset.

Alex Tutor

Dataset · 20 rows

Checkout widget A/B test, one week

20 orders: 10 sessions without the upsell widget (control) and 10 with it (variant). Synthetic data made for this course.

Download CSV

Columns
order_id: Order number
group: control (no widget) or variant (widget shown)
order_value: Order value, USD

Data

order_id	group	order_value
CC-2001	control	38
CC-2002	control	42
CC-2003	control	45
CC-2004	control	40
CC-2005	control	47

Show 15 more rows

CC-2006	control	43
CC-2007	control	41
CC-2008	control	44
CC-2009	control	39
CC-2010	control	46
CC-2051	variant	50
CC-2052	variant	54
CC-2053	variant	47
CC-2054	variant	52
CC-2055	variant	58
CC-2056	variant	49
CC-2057	variant	55
CC-2058	variant	51
CC-2059	variant	53
CC-2060	variant	56

You

That's independent samples, right — the two groups are different customers, not the same person before and after?

Alex Tutor

Right. Independent samples means nothing links a specific control order to a specific variant order. If it were the same customers measured twice, we'd need a paired test instead — that's a later lesson.

Time for a quick check on when this test applies at all.

Alex Tutor

Question 1

Which scenario calls for the classic (independent-samples) t-test?

Choose one answerComparing average order value between two different, unrelated groups of customersComparing the same 50 customers' spend before and after a price changeComparing average satisfaction across four different subscription tiers at onceComparing the proportion of customers who churned in two groups

Alex Tutor

The idea: take the difference between the two means, then divide by how much that difference would naturally wobble from sample to sample — its standard error.

Alex Tutor

t = \frac{\bar{x}_2 - \bar{x}_1}{SE_{\bar{x}_2-\bar{x}_1}}

Student's t-statistic: the difference between the two sample means, divided by the standard error of that difference.

You

And the standard error of a difference is... bigger than the standard error of one group, I'd guess?

Alex Tutor

Good guess. When both groups have similar spread, we pool their variances into one estimate, then combine the two sample sizes into the formula below.

Alex Tutor

SE_{\bar{x}_2-\bar{x}_1} = \sqrt{s_p^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}, \quad s_p^2 = \frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}

Pooled standard error: sp² blends both groups' variances, weighted by their degrees of freedom.

For our data: control has mean

42.50 and s₁ ≈

3.03 (n₁ = 10); variant has mean

52.50 and s₂ ≈

3.37 (n₂ = 10). Your turn to pool them.

Alex Tutor

Question 2

Control: n₁ = 10, mean $42.50, s₁ =$ 3.03. Variant: n₂ = 10, mean $52.50, s₂ =$ 3.37. Using the pooled-variance formula, what is the standard error of the difference between the two means?

Your answer

Units: USD (e.g. 1.25)

With the standard error in hand, the t-statistic itself is one division away.

You

Just to be precise: is 't' itself the thing we called a 'test statistic' back in the intro, or is that a different animal?

Alex Tutor

Same animal. Any single number computed from sample data and used to decide between H0 and H1 is a statistic critical to the test — a test statistic. t is just the one this particular test uses.

You

Difference in means over standard error, like the formula said. Let me plug in your exact SE this time so we don't compound rounding.

Alex Tutor

Question 3

The pooled standard error of the difference is $1.4337. The variant mean is $52.50 and the control mean is $42.50. What is Student's t-statistic?

Your answer

Units: t (e.g. 2.50)

Alex Tutor

One more ingredient before we can judge the t-value: degrees of freedom (df). For the pooled two-sample test, df = n₁ + n₂ - 2 — here, 10 + 10 - 2 = 18.

You

Why subtract 2 and not, say, 1 or 0?

Alex Tutor

Each group's own mean 'uses up' one degree of freedom to estimate that group's variance — two groups, two means used, two degrees lost. It's the same logic from the coin-counting trick you saw with a single sample, just doubled.

With df = 18, a statistics table (or software) gives a critical value of about 2.101 for a two-tailed test at the usual 5% significance level.

Alex Tutor

You

So I compare our t against 2.101 and that tells me if the widget really did something?

Alex Tutor

Question 4

With df = 18 and a critical value of 2.101 at the 5% level, a calculated t of 6.97 means we reject the null hypothesis that the two group means are equal.

TrueFalse

You

What's the 5% actually called, formally? I keep saying '5% cutoff' in meetings and getting funny looks.

Alex Tutor

It's the significance level (α) — рівень значущості, if you want to sound properly official. Pick it before you look at the data, not after.

You

And if the true effect is real but we still fail to reject H0 — say, because our sample was too small — what do we call that mistake?

Alex Tutor

A type II error — помилка другого роду: missing a real effect. Its evil twin, a type I error, is claiming a difference that isn't really there. Match these last few pieces of vocabulary before we move on.

Alex Tutor

Question 5

Match each term to its definition.

Tap an answer, then tap the row it belongs to. You can also drag.

Null hypothesis (H0)
Alternative hypothesis (H1)
Degrees of freedom
Critical value

Answers left to place

Alex Tutor

Two warnings. First: the classic pooled t-test assumes the two groups have roughly equal variances. Ours were close (9.17 vs 11.39) — fine. If they'd been wildly different, we'd use Welch's t-test instead, which adjusts the df downward.

Second, bigger warning: because the formula is built from means, one freakishly large outlier order can wreck it completely — the same problem outliers cause for any mean.

You

What do we do if we suspect outliers, or the data just doesn't look bell-shaped?

Alex Tutor

Then the t-test is the wrong tool, full stop — next lesson covers the rank-based alternative that doesn't care about outliers or shape.

Quick gut-check on assumptions before you leave.

Alex Tutor

Question 6

Which of these are assumptions behind the classic (pooled) Student's t-test for independent samples? Select all that apply.

Choose all that applyThe two samples are independent of each otherEach group's data is roughly normally distributed (or the samples are large enough)The two groups have roughly equal variancesBoth groups must contain exactly the same number of observations

You

Honestly, our t was so far past 2.101 that this feels like overkill for such an obvious result.

Alex Tutor

Let's name-drop properly what we actually ran: the t-критерій Стьюдента для незалежних вибірок (Student's t-test for independent samples), with t-критерій Велча as the fallback for unequal variances. Full names, for your next report.

You

Noted. So even something as ordinary as average order value — sorry, our beloved середній чек — deserved this much rigour, with a sample size (обсяг вибірки) of only 10 per side?

Alex Tutor

Sometimes the data really is that clean. But you only know that because you ran the test — 'looks obvious' has burned plenty of PMs before. Put that instinct into words for me.

Alex Tutor

Question 7Short answer · AI-checked task

Sofia said the result 'feels obvious' and the test seems like overkill. In 2-3 sentences, explain why running the t-test still mattered here, even though the conclusion turned out to be clear-cut.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Nicely argued. Now, a messier version of this exact story, care of the checkout team's group chat.

You

Why do I feel like I'm about to be blamed for something again.

Alex Tutor

Not this time — it's someone else's dataset to critique for once. Small mercy.

You

I'll take it.

Alex Tutor

Question 8Case · AI-checked task

Review the checkout team's claim below.

The checkout team ran their own test on a 'free sample treat' banner: control mean order value $41 (n = 60, s =$ 22), variant mean $46 (n = 60, s =$ 21). They computed t ≈ 1.28 with df = 118, compared it to a critical value of about 1.98 at the 5% level, and posted: 'Banner didn't work, t is below the critical value, killing it.' Is this the right conclusion? What, if anything, would you push back on or want to see before agreeing?

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Mann–Whitney: when data isn't normal