Multifactor ANOVA and interactions

Alex Tutor

Sofia, remember the discount banner test on the product page? This time we split the results by device: Desktop vs Mobile, Control vs Discount banner.

You

So it's basically the same test as before, just with an extra column?

Alex Tutor

More than that. We're testing two factors at once: the banner and the device. Each explanatory variable is called a factor, and together they form a two-way ANOVA.

You

If the banner adds 7 dollars to average revenue overall, it adds 7 dollars everywhere, right?

Alex Tutor

That's exactly the assumption we need to check. Let's look at the four groups this design creates before trusting one overall number.

Alex Tutor

Question 1

Why might looking only at the discount banner's overall average revenue uplift (ignoring device) be misleading?

Choose one answerBecause averages are never reliable for any kind of dataBecause the banner's effect could be very different on desktop vs mobile, and one overall number would hide thatBecause a two-way design cannot compute an overall average at allBecause mobile users never see banners on product pages

Alex Tutor

Session revenue, USD, 3 users per cell: Desktop-Control 38, 40, 42; Desktop-Discount 40, 42, 44; Mobile-Control 28, 30, 32; Mobile-Discount 40, 42, 44.

Alex Tutor

Mean session revenue by banner and deviceThe desktop and mobile lines are not parallel: the discount banner barely moves desktop revenue but lifts mobile revenue much more.

Chart data

Banner	Desktop	Mobile
Control	40	30
Discount	42	42

The four cell means: Desktop-Control 40, Desktop-Discount 42, Mobile-Control 30, Mobile-Discount 42.

Notice the lines aren't parallel. If the banner simply added a fixed amount everywhere, both lines would rise by the same height.

You

The mobile line jumps way more steeply. Is that the interaction thing?

Alex Tutor

Question 2

An interaction effect means that the effect of one factor is the same at every level of the other factor.

TrueFalse

Alex Tutor

A main effect is the average effect of one factor across all levels of the other. Average the banner's effect over both devices.

Averaging desktop and mobile together: Control mean is 35, Discount mean is 42. That's the banner's main effect, ignoring device.

So the banner's main effect is 42 − 35 = 7 dollars, averaged across both devices.

You

And the device factor, does it have a main effect too?

Alex Tutor

Desktop's mean is 41, mobile's is 36, averaged over both banner states. So device's main effect is 5 dollars: desktop earns more regardless of the banner.

Alex Tutor

Question 3

Cell means: Desktop-Control $40, Desktop-Discount$ 42, Mobile-Control $30, Mobile-Discount$ 42. Compute the banner's main effect: the discount mean minus the control mean, averaged over both devices.

Your answer

Units: USD, e.g. 5.0

Alex Tutor

An interaction effect is when one factor's effect depends on the level of the other. It shows up exactly as those non-parallel lines.

On desktop the banner adds 42 − 40 = 2 dollars. On mobile it adds 42 − 30 = 12 dollars. Same banner, very different effect.

You

So mobile users just love discounts more. Let's ship the banner only there, done?

Alex Tutor

That's a fair reading of this data, but check it's not just desktop already sitting near a ceiling before generalising too far.

Alex Tutor

Question 4

Using the same four cell means, compute the interaction effect size: the banner's uplift on mobile (Mobile-Discount − Mobile-Control) minus its uplift on desktop (Desktop-Discount − Desktop-Control).

Your answer

Units: USD, e.g. 5.0

Alex Tutor

F_{banner} = \dfrac{MS_{banner}}{MS_{error}}, \quad F_{device} = \dfrac{MS_{device}}{MS_{error}}, \quad F_{banner \times device} = \dfrac{MS_{banner \times device}}{MS_{error}}

A two-way ANOVA computes three separate F-statistics: one per main effect, plus one for their interaction.

For this test: F_banner ≈ 36.75, F_device ≈ 18.75, F_interaction ≈ 18.75, all on df(1, 8). All three clear the critical value of about 5.32.

You

All three significant? So everything matters equally?

Alex Tutor

They matter, but the interaction is the headline: it changes how you should even read the two main effects.

Alex Tutor

Question 5

Match each term from a multifactor ANOVA to what it describes.

Tap an answer, then tap the row it belongs to. You can also drag.

Factor
Main effect
Interaction effect
Two-way (multifactor) ANOVA

Answers left to place

Alex Tutor

With a significant interaction, one 'roll out everywhere or nowhere' decision is the wrong shape for the question.

Treat a significant interaction as a flag to segment the decision: it's plausible to ship on mobile while holding or testing further on desktop.

You

Feels like Simpson's paradox's cousin.

Alex Tutor

Close: an unnoticed interaction is exactly how a reassuring overall average can hide a segment where the change isn't helping much at all.

Alex Tutor

You

So even more testing after the testing?

Alex Tutor

Exactly the same discipline as ANOVA's post hoc step, just applied one layer deeper, to combinations instead of single groups.

Alex Tutor

Question 6

Which statements about factorial designs and interactions are true? Select all that apply.

Choose all that applyA significant interaction means the two main effects should be interpreted with cautionNon-parallel lines in a chart of cell means are a visual sign of a possible interactionA two-way ANOVA can only detect main effects, never an interactionIf the interaction is significant, one overall rollout decision may be wrong for some segmentsMain effects and the interaction are always tested with the exact same F-statistic value

Alex Tutor

One banner, two very different stories: +2 dollars on desktop, +12 on mobile. A single 'it works' verdict erases that gap.

Whenever a chart of group means shows crossing or clearly non-parallel lines, that's your cue to look for an interaction before reporting one overall number.

You

Got it: check the lines before trusting the average.

Alex Tutor

One task to put that into your own words, then a trickier case with three channels instead of two devices.

Alex Tutor

Question 7Short answer · AI-checked task

In 2–4 sentences, explain why CatChow shouldn't just look at the discount banner's overall average uplift (+7 dollars) when deciding whether to roll it out on both desktop and mobile.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Nice. One more, mixing both main effects and an interaction.

This one has no chart, just a results table. Read it like an analyst would, one channel at a time.

You

No pressure.

Alex Tutor

You'll be fine, that's exactly what this course is for.

Alex Tutor

Question 8Case · AI-checked task

What would you recommend, channel by channel?

CatChow tested a loyalty-points banner (off vs on) across three acquisition channels (Search, Social, Email), measuring average order value. Cell means: Search off $32 / on$ 34; Social off $28 / on$ 40; Email off $45 / on$ 46. A two-way ANOVA finds the banner × channel interaction significant (F = 9.2, p = 0.002). The overall average uplift across all three channels is about +5 dollars, and marketing wants to enable the banner site-wide because that average is positive.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Paired tests: before and after