How to investigate a sudden spike in a metric

Maya

Monday morning, and the dashboard says sign-ups over the long weekend were up 156%: 2,014 in three days. It's the new Android sign-up screen, isn't it? I want to tell the team.

Alex Tutor

Maybe. Before anyone celebrates, list everything that could make a spike like this. Four families: tracking, traffic, product and outside events.

You

Tracking: we count something twice. Traffic: a campaign or a store placement. Product: a release changed what people do. Outside: a holiday, the news, a competitor's outage.

Alex Tutor

Then rule them out one by one with the data we already have, cheapest check first. That's a spike investigation. Here's the change log for the period.

Alex Tutor

What changed, 29 August – 18 September

Date	What changed
Thu 10 Sep	Android `5.2` starts rolling out: a new phone-number sign-up screen (about 30% of Android sign-ups that day, 70% on the 11th, all from the 12th)
Wed 16 Sep	Android `5.2.1` with bug fixes, from midday
Whole period	Paid social and store search ads budgets unchanged; no store featuring; no iOS release
Sat 12 – Mon 14 Sep	Public-holiday long weekend in several of Halves's biggest markets

Alex Tutor

Question 1

Match each family of causes with the first check you'd run on this data.

Pick an answer for each row. Choosing an answer that is already used moves it to this row.

Tracking
Choose an answer
Traffic
Choose an answer
Product
Choose an answer
Outside events
Choose an answer

Alex Tutor

"Up 156%" needs a baseline. Weekends and Mondays differ from midweek, so compare each spike day with the same weekday in the two weeks before, and average those two.

You

So Saturday the 12th against Saturdays the 29th and the 5th, and the same for Sunday and Monday.

Alex Tutor

Yes. And count two things side by side: signup_events, which the dashboard shows, and new_users, distinct new accounts.

Alex Tutor

Dataset · 168 rows

Halves sign-ups by day, platform and channel

Saturday 29 August – Friday 18 September 2026; 21 days × 2 platforms × 4 channels.

Download CSV

Columns
date: Day (UTC), Saturday 29 August – Friday 18 September 2026
platform: ios or android
acquisition_channel: invite, app_store_search, paid_social or store_search_ads
signup_events: Number of `signed_up` events: what the dashboard calls "sign-ups"
new_users: Distinct new accounts created that day
new_users_in_trip_groups: Of the new users, those whose first group is a trip group

Data

date	platform	acquisition_channel	signup_events	new_users	new_users_in_trip_groups
2026-08-29	ios	invite	71	71	11
2026-08-29	android	invite	55	55	9
2026-08-29	ios	app_store_search	34	34	2
2026-08-29	android	app_store_search	25	25	1
2026-08-29	ios	paid_social	35	35	2

Show 163 more rows

2026-08-29	android	paid_social	27	27	1
2026-08-29	ios	store_search_ads	18	18	1
2026-08-29	android	store_search_ads	14	14	1
2026-08-30	ios	invite	65	65	11
2026-08-30	android	invite	58	58	8
2026-08-30	ios	app_store_search	32	32	1
2026-08-30	android	app_store_search	26	26	1
2026-08-30	ios	paid_social	34	34	2
2026-08-30	android	paid_social	26	26	1
2026-08-30	ios	store_search_ads	17	17	1
2026-08-30	android	store_search_ads	14	14	1
2026-08-31	ios	invite	56	56	9
2026-08-31	android	invite	48	48	7
2026-08-31	ios	app_store_search	29	29	2
2026-08-31	android	app_store_search	22	22	1
2026-08-31	ios	paid_social	32	32	2
2026-08-31	android	paid_social	27	27	1
2026-08-31	ios	store_search_ads	17	17	1
2026-08-31	android	store_search_ads	13	13	1
2026-09-01	ios	invite	54	54	8
2026-09-01	android	invite	46	46	7
2026-09-01	ios	app_store_search	31	31	2
2026-09-01	android	app_store_search	22	22	1
2026-09-01	ios	paid_social	32	32	1
2026-09-01	android	paid_social	27	27	1
2026-09-01	ios	store_search_ads	16	16	1
2026-09-01	android	store_search_ads	15	15	1
2026-09-02	ios	invite	53	53	9
2026-09-02	android	invite	47	47	7
2026-09-02	ios	app_store_search	31	31	2
2026-09-02	android	app_store_search	22	22	1
2026-09-02	ios	paid_social	31	31	2
2026-09-02	android	paid_social	27	27	1
2026-09-02	ios	store_search_ads	18	18	1
2026-09-02	android	store_search_ads	14	14	1
2026-09-03	ios	invite	59	59	9
2026-09-03	android	invite	47	47	7
2026-09-03	ios	app_store_search	28	28	1
2026-09-03	android	app_store_search	23	23	1
2026-09-03	ios	paid_social	36	36	2
2026-09-03	android	paid_social	28	28	1
2026-09-03	ios	store_search_ads	18	18	1
2026-09-03	android	store_search_ads	15	15	1
2026-09-04	ios	invite	58	58	9
2026-09-04	android	invite	46	46	7
2026-09-04	ios	app_store_search	27	27	1
2026-09-04	android	app_store_search	23	23	1
2026-09-04	ios	paid_social	31	31	2
2026-09-04	android	paid_social	27	27	1
2026-09-04	ios	store_search_ads	18	18	1
2026-09-04	android	store_search_ads	13	13	1
2026-09-05	ios	invite	63	63	10
2026-09-05	android	invite	50	50	8
2026-09-05	ios	app_store_search	33	33	2
2026-09-05	android	app_store_search	24	24	1
2026-09-05	ios	paid_social	31	31	2
2026-09-05	android	paid_social	27	27	1
2026-09-05	ios	store_search_ads	17	17	1
2026-09-05	android	store_search_ads	15	15	1
2026-09-06	ios	invite	71	71	11
2026-09-06	android	invite	53	53	8
2026-09-06	ios	app_store_search	30	30	2
2026-09-06	android	app_store_search	27	27	1
2026-09-06	ios	paid_social	34	34	2
2026-09-06	android	paid_social	26	26	1
2026-09-06	ios	store_search_ads	17	17	1
2026-09-06	android	store_search_ads	14	14	1
2026-09-07	ios	invite	56	56	8
2026-09-07	android	invite	47	47	7
2026-09-07	ios	app_store_search	30	30	2
2026-09-07	android	app_store_search	23	23	1
2026-09-07	ios	paid_social	33	33	2
2026-09-07	android	paid_social	26	26	1
2026-09-07	ios	store_search_ads	16	16	1
2026-09-07	android	store_search_ads	13	13	1
2026-09-08	ios	invite	52	52	8
2026-09-08	android	invite	47	47	7
2026-09-08	ios	app_store_search	28	28	1
2026-09-08	android	app_store_search	24	24	1
2026-09-08	ios	paid_social	32	32	2
2026-09-08	android	paid_social	28	28	1
2026-09-08	ios	store_search_ads	17	17	1
2026-09-08	android	store_search_ads	13	13	1
2026-09-09	ios	invite	52	52	8
2026-09-09	android	invite	47	47	7
2026-09-09	ios	app_store_search	30	30	2
2026-09-09	android	app_store_search	24	24	1
2026-09-09	ios	paid_social	33	33	2
2026-09-09	android	paid_social	27	27	1
2026-09-09	ios	store_search_ads	18	18	1
2026-09-09	android	store_search_ads	14	14	1
2026-09-10	ios	invite	56	56	9
2026-09-10	android	invite	55	42	6
2026-09-10	ios	app_store_search	29	29	2
2026-09-10	android	app_store_search	31	24	1
2026-09-10	ios	paid_social	31	31	1
2026-09-10	android	paid_social	34	26	1
2026-09-10	ios	store_search_ads	16	16	1
2026-09-10	android	store_search_ads	18	14	1
2026-09-11	ios	invite	59	59	10
2026-09-11	android	invite	80	47	7
2026-09-11	ios	app_store_search	30	30	1
2026-09-11	android	app_store_search	37	22	1
2026-09-11	ios	paid_social	33	33	2
2026-09-11	android	paid_social	43	25	1
2026-09-11	ios	store_search_ads	16	16	1
2026-09-11	android	store_search_ads	22	13	1
2026-09-12	ios	invite	177	177	104
2026-09-12	android	invite	278	139	86
2026-09-12	ios	app_store_search	45	45	10
2026-09-12	android	app_store_search	68	34	6
2026-09-12	ios	paid_social	33	33	6
2026-09-12	android	paid_social	54	27	5
2026-09-12	ios	store_search_ads	16	16	3
2026-09-12	android	store_search_ads	28	14	3
2026-09-13	ios	invite	168	168	111
2026-09-13	android	invite	292	146	93
2026-09-13	ios	app_store_search	45	45	10
2026-09-13	android	app_store_search	66	33	7
2026-09-13	ios	paid_social	31	31	6
2026-09-13	android	paid_social	52	26	6
2026-09-13	ios	store_search_ads	16	16	3
2026-09-13	android	store_search_ads	26	13	3
2026-09-14	ios	invite	154	154	97
2026-09-14	android	invite	236	118	65
2026-09-14	ios	app_store_search	37	37	8
2026-09-14	android	app_store_search	62	31	6
2026-09-14	ios	paid_social	35	35	7
2026-09-14	android	paid_social	50	25	5
2026-09-14	ios	store_search_ads	17	17	4
2026-09-14	android	store_search_ads	28	14	3
2026-09-15	ios	invite	52	52	9
2026-09-15	android	invite	90	45	7
2026-09-15	ios	app_store_search	30	30	1
2026-09-15	android	app_store_search	44	22	1
2026-09-15	ios	paid_social	33	33	2
2026-09-15	android	paid_social	58	29	2
2026-09-15	ios	store_search_ads	16	16	1
2026-09-15	android	store_search_ads	28	14	1
2026-09-16	ios	invite	57	57	9
2026-09-16	android	invite	66	44	6
2026-09-16	ios	app_store_search	28	28	1
2026-09-16	android	app_store_search	33	22	1
2026-09-16	ios	paid_social	31	31	2
2026-09-16	android	paid_social	41	27	1
2026-09-16	ios	store_search_ads	17	17	1
2026-09-16	android	store_search_ads	21	14	1
2026-09-17	ios	invite	57	57	8
2026-09-17	android	invite	46	46	7
2026-09-17	ios	app_store_search	29	29	1
2026-09-17	android	app_store_search	23	23	1
2026-09-17	ios	paid_social	32	32	1
2026-09-17	android	paid_social	25	25	1
2026-09-17	ios	store_search_ads	17	17	1
2026-09-17	android	store_search_ads	14	14	1
2026-09-18	ios	invite	53	53	8
2026-09-18	android	invite	45	45	6
2026-09-18	ios	app_store_search	28	28	1
2026-09-18	android	app_store_search	25	25	1
2026-09-18	ios	paid_social	34	34	2
2026-09-18	android	paid_social	26	26	1
2026-09-18	ios	store_search_ads	18	18	1
2026-09-18	android	store_search_ads	15	15	1

The baseline for the three days is 785.5 in both columns. Now compare the spike days with it.

Alex Tutor

Question 2

By how much did distinct new users on 12–14 September exceed the baseline (the average of the same weekdays in the two previous weeks)? Give the change in percent.

Your answer

Units: %, e.g. 12.5

You

Events went up 156% but new accounts only 77%. So 620 events on those days aren't new people.

Alex Tutor

Split by platform. On iOS it's 1.00 event per new user, as always. On Android, 2.00 on all three days.

You

And the Android extras start on the 10th, the day 5.2 began rolling out, and stop after the fix on the 16th.

Alex Tutor

The new screen sends signed_up twice. That's half of Maya's excess: a counting bug, not people.

Alex Tutor

Question 3

On 12–14 September Android shows 2.00 signed_up events per new user and iOS 1.00; Android's extra events start on 10 September and stop after 16 September. What follows?

Choose one answerAndroid users sign up twice as eagerly on the new screenAndroid 5.2 logs each sign-up twice, so about half of the excess events are duplicatesiOS tracking is broken and misses half of its sign-upsThe spike is entirely a tracking error

Maya

Fine, but 608 real extra users is still a lot. Where did they come from?

Alex Tutor

By channel: invites +555.5, store search +57.5, paid social −2, store search ads −2.5. No campaign did this, and budgets didn't change.

You

And by platform: iOS new users +77.7%, Android +77.1%. If the Android screen had helped, Android would rise more.

Alex Tutor

So the release explains the extra events, not the extra people.

Maya

So the new screen did nothing at all?

Alex Tutor

Nothing we can see in sign-ups. Whether it's easier to use is a separate question, and an A/B test, not a spike, can answer it.

Alex Tutor

Question 4

Which explanations of the real rise in new users can you rule out with this data? Select all that apply.

Choose all that applyA paid campaign brought themThe new Android sign-up screen converted more visitorsGroups formed for trips pulled in new members through invitesThe rise is only the duplicate events

Theo

The 12th to the 14th was a long holiday weekend in several of our biggest markets. People go away in groups.

Alex Tutor

And the data agrees: 47% of spike-day new users started in a trip group, against 10% on the baseline days. That's consistent with the holiday, though a comparison like this can't prove it.

You

Then these users may behave like the summer cohorts and leave after the trip.

Alex Tutor

Right, so nobody should raise the autumn forecast on this weekend. And the dashboard needs fixing so the next spike isn't half fake.

Alex Tutor

Question 5Short answer · AI-checked task

Fix the dashboard in 3–5 sentences: rewrite the definition of "sign-ups" so this can't happen again, and describe one automatic check that would have caught the Android bug on 10 September.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Maya

OK. I need something I can forward to the team today, before the rumor about the new screen spreads.

Alex Tutor

Lead with the answer, then the checks, with numbers: what was ruled out, what's left, how sure we are, and what we're fixing.

Alex Tutor

Question 6Case · AI-checked task

Write the memo to Maya: what caused the sign-up spike on 12–14 September?

Maya's dashboard shows sign-ups (the count of signed_up events) for Saturday 12 – Monday 14 September at 2,014, +156% on the usual level. She believed the new Android sign-up screen caused it and wants a memo she can forward to the team today.

You have the daily sign-up data above and the change log:

Date	What changed
Thu 10 Sep	Android `5.2` starts rolling out: a new phone-number sign-up screen (about 30% of Android sign-ups that day, 70% on the 11th, all from the 12th)
Wed 16 Sep	Android `5.2.1` with bug fixes, from midday
Whole period	Paid social and store search ads budgets unchanged; no store featuring; no iOS release
Sat 12 – Mon 14 Sep	Public-holiday long weekend in several of Halves's biggest markets

Write 200–350 words:

The answer first: what happened, in one or two sentences.
The hypotheses you considered (tracking, traffic, product, outside) and how the data rules each in or out, with numbers against a stated baseline.
How sure you are, and what the data can't prove.
What to do now: the dashboard, and what this means for the autumn forecast.

Dataset for this task: Halves sign-ups by day, platform and channel

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

The habit to keep: a spike is a question, not a result. List the families, compare with a baseline, check tracking first, and say how sure you are.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Back to the course