Correlation vs causation in product analytics

Maya

Look what I found for the Plus pitch. New users who scan a receipt in their first week are active in week 4 at 49%. Everyone else: 27%. Let's push scanning in onboarding and lift retention for everyone.

Theo

I'd put it on the store page too: “Scanners stay longer.”

Alex Tutor

The numbers are right. The risky part is the jump from “people who scan retain better” to “scanning makes people retain”. The first is a correlation. The second is a claim about cause.

You

What else could make scanners stay?

Alex Tutor

Ask who scans. Free users get three scans a month; you scan when you're the one paying for dinners and groceries. Here are April–May sign-ups who added at least one expense in week 1, split by how many they added.

Alex Tutor

Dataset · 6 rows

Receipt scanning and week-4 retention

New Halves users from April–May 2026 who added at least one expense in days 0–6.

Download CSV

Columns
week1_expenses: Expenses added in days 0–6: 1, 2_to_4 or 5_plus
scanned_receipt_week1: Scanned at least one receipt in days 0–6: yes or no
users: Users in this row
active_days_22_28: Of them, opened the app on at least one of days 22–28

Data

week1_expenses	scanned_receipt_week1	users	active_days_22_28
1	yes	57	9
1	no	1343	173
2_to_4	yes	133	49
2_to_4	no	967	335
5_plus	yes	303	184

Show 1 more row

5_plus

no

397

233

Alex Tutor

Question 1

Check Maya's gap: week-4 retention of scanners minus that of non-scanners, all buckets together. Percentage points, two decimals.

Your answer

Units: pp

Alex Tutor

Now look at the rows, not the total. Where do the scanners sit?

You

Mostly in 5_plus: 303 of 493. Among non-scanners, 5_plus is only 397 of 2,707.

Alex Tutor

So scanners were already the busiest users in week 1. That's selection bias: the two groups differ because of who ended up in each, not only because of the scanning.

You

Heavy users retain better anyway, whether they scan or not.

Alex Tutor

Right, and the slide compares a group that's mostly heavy users with a group that's mostly light ones. Check the share yourself.

Alex Tutor

Question 2

What share of scanners are in the 5_plus bucket? Percent, two decimals. (Among non-scanners it's 14.67%.)

Your answer

Units: %

Alex Tutor

So compare scanners with non-scanners who were equally busy: inside each bucket.

Maya

Start with the heavy users, that's where most scanners are.

You

Same activity, different receipts. If scanning itself mattered a lot, the gap should survive inside each bucket.

Alex Tutor

Question 3

In the 5_plus bucket, what is the week-4 retention gap, scanners minus non-scanners? Percentage points, two decimals.

Your answer

Units: pp

Two to three points in every bucket, not 22. If we give scanners the non-scanners' mix of buckets, their retention is 29.9% against 27.4%: about +2.5 pp.

Maya

So scanning does nothing?

Alex Tutor

Not that either. Some gap remains, but we split by one thing only. Scanners may also be the group's treasurer, travel more or be on a Plus trial. Observational data can't pull those apart from scanning itself.

You

So it can say “scanners retain better, mostly because they're busier”, but not “scanning makes people stay”.

Alex Tutor

Question 4

Which statements does the receipt-scanning data support? Select all that apply.

Choose all that applyNew users who scanned a receipt in week 1 retained better in week 4 than those who didn'tScanning a receipt raises week-4 retention by about 22 pointsMost of the gap comes from scanners being busier in week 1If every new user scans a receipt, retention will reach 49%Among equally busy users, scanners retained 2–3 points better

Theo

Then how do we ever know if pushing scanning is worth it?

Alex Tutor

Test the thing you'd actually ship: an onboarding prompt that suggests scanning. Randomize who gets it, and compare everyone assigned, scanners or not. Comparing only those who scanned would bring the selection right back.

You

And the answer might be much smaller than 22 points.

Alex Tutor

Likely. A prompt only nudges some people to scan, so its effect on everyone assigned is smaller still. Size the test for that, not for the slide.

Alex Tutor

Question 5

What is the best way to learn whether promoting receipt scanning raises week-4 retention?

Choose one answerRandomize an onboarding prompt that suggests scanning, and compare everyone who was assigned the prompt with everyone who wasn'tCompare users who scanned with users who didn't, over a longer periodShow the prompt to everyone and compare scanners before and after the changeRandomize the prompt, then compare only the users who scanned in each variant

Maya

Fine. But I still want the slide. Help me word it so it's true.

Alex Tutor

Say what the data shows, name the selection, and state what you'd test. That's a stronger slide than a claim someone can knock down with one question.

You

And keep the 49% and 27%, just not the arrow between them.

Alex Tutor

Question 6Short answer · AI-checked task

Rewrite Maya's slide claim “Scanning receipts boosts retention by 22 points” in 2–4 sentences, so that it is true to the data and useful for the Plus pitch.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Case: the version or the traffic?