Did the release help, or did the traffic change?

Maya

5.2.0 shipped the new onboarding on 31 August. Week-2 retention of new users went from 28.9% in August to 39.8% in September. Eleven points! That's going in the board update.

Theo

Also worth knowing: I paused the paid social campaign on 1 September. The summer budget ran out.

Alex Tutor

And September is moving season: new flatmates, new rent to split, while summer trips wind down. Three things changed in the same week. So we check in order: is the number fair, who came, and what like for like shows.

Alex Tutor

Dataset · 72 rows

Weekly sign-up cohorts, August and September

New Halves users, weeks of 3 August to 21 September 2026. Version 5.2.0 with the new onboarding was released on 31 August. Every cohort has its full days 8–14.

Download CSV

Columns
signup_week: Monday of the sign-up week
app_version: Version installed: 5.1.0 in August, 5.2.0 from 31 August
channel: invite, app_store_search or paid_social
first_group_type: Type of the first group the user joined: flat, couple or trip
new_users: New users in this cell
active_days_8_14: Of them, opened the app on at least one of days 8–14

Data

signup_week	app_version	channel	first_group_type	new_users	active_days_8_14
2026-08-03	5.1.0	invite	flat	71	29
2026-08-03	5.1.0	invite	couple	60	30
2026-08-03	5.1.0	invite	trip	102	17
2026-08-03	5.1.0	app_store_search	flat	51	19
2026-08-03	5.1.0	app_store_search	couple	43	14

Show 67 more rows

2026-08-03	5.1.0	app_store_search	trip	75	9
2026-08-03	5.1.0	paid_social	flat	64	14
2026-08-03	5.1.0	paid_social	couple	53	12
2026-08-03	5.1.0	paid_social	trip	90	10
2026-08-10	5.1.0	invite	flat	71	37
2026-08-10	5.1.0	invite	couple	59	23
2026-08-10	5.1.0	invite	trip	101	17
2026-08-10	5.1.0	app_store_search	flat	51	24
2026-08-10	5.1.0	app_store_search	couple	42	17
2026-08-10	5.1.0	app_store_search	trip	81	14
2026-08-10	5.1.0	paid_social	flat	62	21
2026-08-10	5.1.0	paid_social	couple	53	15
2026-08-10	5.1.0	paid_social	trip	91	6
2026-08-17	5.1.0	invite	flat	69	30
2026-08-17	5.1.0	invite	couple	58	23
2026-08-17	5.1.0	invite	trip	104	12
2026-08-17	5.1.0	app_store_search	flat	49	18
2026-08-17	5.1.0	app_store_search	couple	42	16
2026-08-17	5.1.0	app_store_search	trip	75	16
2026-08-17	5.1.0	paid_social	flat	61	20
2026-08-17	5.1.0	paid_social	couple	55	17
2026-08-17	5.1.0	paid_social	trip	94	11
2026-08-24	5.1.0	invite	flat	68	35
2026-08-24	5.1.0	invite	couple	60	25
2026-08-24	5.1.0	invite	trip	101	21
2026-08-24	5.1.0	app_store_search	flat	53	27
2026-08-24	5.1.0	app_store_search	couple	45	20
2026-08-24	5.1.0	app_store_search	trip	78	18
2026-08-24	5.1.0	paid_social	flat	61	29
2026-08-24	5.1.0	paid_social	couple	53	25
2026-08-24	5.1.0	paid_social	trip	95	14
2026-08-31	5.2.0	invite	flat	167	79
2026-08-31	5.2.0	invite	couple	69	29
2026-08-31	5.2.0	invite	trip	25	4
2026-08-31	5.2.0	app_store_search	flat	116	48
2026-08-31	5.2.0	app_store_search	couple	51	14
2026-08-31	5.2.0	app_store_search	trip	19	2
2026-08-31	5.2.0	paid_social	flat	38	9
2026-08-31	5.2.0	paid_social	couple	17	5
2026-08-31	5.2.0	paid_social	trip	6	1
2026-09-07	5.2.0	invite	flat	164	92
2026-09-07	5.2.0	invite	couple	71	30
2026-09-07	5.2.0	invite	trip	25	9
2026-09-07	5.2.0	app_store_search	flat	118	51
2026-09-07	5.2.0	app_store_search	couple	52	19
2026-09-07	5.2.0	app_store_search	trip	18	2
2026-09-07	5.2.0	paid_social	flat	39	10
2026-09-07	5.2.0	paid_social	couple	17	7
2026-09-07	5.2.0	paid_social	trip	6	0
2026-09-14	5.2.0	invite	flat	155	82
2026-09-14	5.2.0	invite	couple	72	33
2026-09-14	5.2.0	invite	trip	26	6
2026-09-14	5.2.0	app_store_search	flat	118	40
2026-09-14	5.2.0	app_store_search	couple	53	17
2026-09-14	5.2.0	app_store_search	trip	20	4
2026-09-14	5.2.0	paid_social	flat	36	15
2026-09-14	5.2.0	paid_social	couple	17	5
2026-09-14	5.2.0	paid_social	trip	6	0
2026-09-21	5.2.0	invite	flat	154	69
2026-09-21	5.2.0	invite	couple	71	27
2026-09-21	5.2.0	invite	trip	25	2
2026-09-21	5.2.0	app_store_search	flat	122	58
2026-09-21	5.2.0	app_store_search	couple	54	19
2026-09-21	5.2.0	app_store_search	trip	19	1
2026-09-21	5.2.0	paid_social	flat	38	11
2026-09-21	5.2.0	paid_social	couple	18	4
2026-09-21	5.2.0	paid_social	trip	6	3

You

The fairness rules pass: new users by sign-up week, a fixed days 8–14, every cohort complete.

Alex Tutor

Good. Then reproduce Maya's before number first, so we know we're reading the same data.

Alex Tutor

Question 1

What share of 5.1.0 (August) new users were active in days 8–14? Percent, two decimals.

Your answer

Units: %

Maya

So my number is right.

Alex Tutor

The number is right; the question is what it's a number of. Who are these people? In August, 45% of new users joined through a trip group and 34% came from paid social. Both groups retain worst.

You

And in September trips shrink and paid stops. What about flats?

Alex Tutor

Check.

Alex Tutor

Question 2

What share of 5.2.0 (September) new users had flat as their first group type? Percent, two decimals. (August: 29.95%.)

Your answer

Units: %

Alex Tutor

So September is full of the users who always retain best. Now compare inside each group type.

Maya

Flats went from 41.5% to 44.6%. That's still up!

Alex Tutor

Look one level down. September's flats came more through invites, the best channel: invite flats retain around 47–50%, paid flats around 30–34%. Group type alone still hides a channel shift.

Alex Tutor

Question 3

What share of 5.2.0 couple users was active in days 8–14? Percent, two decimals. (5.1.0 couples: 38.04%.)

Your answer

Units: %

One summary number, as before: September's rate in each channel × group type cell, weighted by August's mix. It comes to 28.82%, against 28.88% in August: −0.06 points.

Theo

How can flats go up and the whole thing stay flat?

Alex Tutor

Because inside flats the channel mix moved too: more invites. Re-weighting by channel and group type together fixes both at once, cell by cell.

You

So the whole eleven points is who signed up.

Alex Tutor

Question 4

What best explains the jump from 28.9% to 39.8%?

Choose one answerThe mix changed: more flats and invites, fewer trips and paid users; like for like, retention barely movedThe new onboarding in 5.2.0 raised week-2 retention by about 11 pointsPausing paid social made the remaining users more loyalIncomplete September cohorts inflate the after number

Theo

Then my pause did it! Cutting paid social raised retention by eleven points.

Alex Tutor

It raised the average. Nobody retained better because of it. And the question for paid social was never its retention alone; it's what its users are worth against what they cost.

You

Same trap, different hero.

Alex Tutor

Question 5Short answer · AI-checked task

Theo says: “Pausing paid social raised retention.” Reply to him in 2–4 sentences.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Maya

So what do I tell the board? I still think the onboarding is better.

You

Could a holdout have answered it?

Alex Tutor

Yes. A random slice of new users kept on the old onboarding lives through the same September, with the same flats and the same paused campaign. Then the difference is the onboarding.

It might be, by a little. This data can't show a small effect either way. Write what it does show, what it can't, and how we'd find out properly next time.

Alex Tutor

Question 6Case · AI-checked task

Write the board update: did 5.2.0 help, or did the traffic change?

Version 5.2.0 of Halves, with a new onboarding flow, was released on Monday, 31 August 2026. The same week, the paid social campaign was paused, and September's moving season began. The dataset above has weekly cohorts of new users from 3 August to 21 September by channel and first group type; every cohort has its full days 8–14.

Maya plans to tell the board: “The new onboarding lifted week-2 retention from 28.9% to 39.8%.”

Write the paragraph she should send instead (200–300 words). Cover:

Whether the raw comparison is built fairly, and its numbers.
What changed in who signed up, with numbers.
A like-for-like comparison (within segments and/or re-weighted to August's mix) and what it shows.
What this comparison can and can't conclude about the new onboarding.
What the team should do to learn the onboarding's real effect.

Dataset for this task: Weekly sign-up cohorts, August and September

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

That's the whole module in one paragraph: build the number fairly, ask who came, compare like with like, and hand the causal question to a test.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Revenue metrics