Simpson's paradox
A trend that appears in every subgroup reverses when the groups are combined, because group sizes differ.
Simpson's paradox is when a comparison that holds in every subgroup reverses, or disappears, once you pool the subgroups together. It isn't an arithmetic error. An overall conversion rate is a weighted average of the segment rates, and the weights (the segment mix) can differ between the groups you're comparing. If one group gets most of its traffic from a low-converting segment, its overall rate is dragged down however well it does inside each segment.
The segment variable is acting as a confounder: it's related both to which group a user belongs to and to the outcome. In product and marketing work the paradox tends to show up when you:
- compare channels whose audiences differ (mostly new vs mostly returning visitors);
- compare before and after a release while the traffic mix shifted (say, a growing share of mobile users);
- analyse features people opt into, where heavy users self-select;
- pool several countries or campaigns with very different sizes;
- change the traffic split halfway through an A/B test and then combine the periods.
Which number should you trust? That depends on the causal story, not on the arithmetic. If the segment influences the outcome and isn't caused by the thing you're comparing, compare within segments, or reweight everyone to a common mix. If the segment is itself a result of the treatment (a feature that pushes people into becoming subscribers, for example), splitting by it can mislead instead.
Proper randomisation is the built-in defence. A clean A/B test keeps segment mixes balanced on average, so a Simpson-style reversal inside a randomised test is a red flag for a sample ratio mismatch or a mid-test allocation change.
Example
CatChow's marketing team compares two channels over one month: the email newsletter and Instagram ads, 1,000 site visits each.
Instagram converts better in both segments (13.0% vs 12.0% and 2.5% vs 2.0%) yet looks less than half as good overall (4.6% vs 10.0%).
Why: the totals are weighted averages with very different weights. 80% of email visits come from returning customers; 80% of Instagram visits come from new visitors, who rarely buy on a first visit.
- Email:
- Instagram:
Reweight both channels to the same 50/50 mix and the order flips back:
- Email:
- Instagram:
So which channel is "better"? For "which channel converts a given kind of visitor better?", Instagram. For "which channel produces more orders per 1,000 visits today?", email. They're different questions, and the aggregate table on its own answers only the second.
Common mistakes
- Judging channels, cohorts or releases by the aggregate rate alone. Always check whether the segment mix differs between the things you compare.
- Blaming a release for a drop that's really a mix shift. If mobile traffic grew from 30% to 60% of visits, overall conversion can fall even though both mobile and desktop improved.
- Fishing for a reversal. Slice by enough segments and some comparison will flip by chance. Small segments are noisy, and many slices bring the multiple comparisons problem.
- Always trusting the segmented view. If the treatment itself changes which segment people end up in, splitting by that segment compares unlike groups. Decide based on the causal story.
- Pooling A/B test periods with different traffic splits. Moving from 10/90 to 50/50 mid-test and adding the numbers together can manufacture a reversal; analyse each period separately or weight them properly.
Learn it in the course
- Multifactor ANOVA and interactions · When two factors act together, e.g. a promo that works on mobile but not on desktop.