How to read A/B test results: effect, interval, decision

Maya

The paywall test is done! I'm putting “Trials +74%” in the all-hands deck.

Theo

And iOS alone is up 2 points. Can we ship there first?

Alex Tutor

Let's read it in the order the plan fixed six weeks ago: check the split, then the primary metric with its interval, then the guardrail, then the rule. Trials and platforms come last.

Alex Tutor

Dataset · 16 rows

Halves paywall test: control vs trial_first

Users enrolled at their first paywall view, 3–30 August 2026, each followed for 14 days. One row per enrollment week, platform and variant.

Download CSV

Columns
enroll_week: Monday of the week the users entered the test
platform: ios or android
variant: `control` (current paywall) or `trial_first` (leads with a 7-day trial)
users: Users enrolled
trials_started: Users who started a trial within 14 days (secondary)
paid_14d: Users with a first paid charge within 14 days (primary)
active_days_8_14: Users who added an expense in days 8–14 (guardrail)

Data

enroll_week	platform	variant	users	trials_started	paid_14d	active_days_8_14
2026-08-03	ios	control	1002	82	43	623
2026-08-03	ios	trial_first	974	161	59	601
2026-08-03	android	control	715	66	27	429
2026-08-03	android	trial_first	720	108	33	430
2026-08-10	ios	control	1009	88	39	612

Show 11 more rows

2026-08-10	ios	trial_first	963	117	57	582
2026-08-10	android	control	714	60	23	398
2026-08-10	android	trial_first	681	90	26	420
2026-08-17	ios	control	1041	82	39	632
2026-08-17	ios	trial_first	1027	163	61	611
2026-08-17	android	control	709	57	27	423
2026-08-17	android	trial_first	654	117	24	408
2026-08-24	ios	control	992	98	40	634
2026-08-24	ios	trial_first	1065	150	64	636
2026-08-24	android	control	715	54	22	424
2026-08-24	android	trial_first	708	98	21	428

The split first. 6,897 users in control, 6,792 in trial_first. With a 50/50 assignment that gap is normal; a chi-square check gives 0.81, well under 3.84. A big imbalance would mean broken assignment, and then nothing else is worth reading.

You

What would a broken split look like?

Alex Tutor

Something like 7,400 against 6,300. It's usually a bug: one variant crashes on older phones, or logs some users twice. You fix it and rerun; you don't interpret it.

You

Then the primary: sum paid_14d and users for each variant and divide.

Alex Tutor

Question 1

Use the dataset. What share of control users paid within 14 days? Answer in percent, to two decimals.

Your answer

Units: %

Alex Tutor

trial_first, same sums: 345 of 6,792, so 5.08%.

You

So is the effect 5.08 − 3.77, or 5.08 ÷ 3.77?

Alex Tutor

The effect is always treatment − control, in percentage points. The ratio is a relative lift, and it must be labelled as one.

Theo

Points sound smaller than percent.

Alex Tutor

Which is exactly why a readout says which one it means.

Alex Tutor

Question 2

What is the effect on the primary metric, treatment − control, in percentage points? Two decimals.

Your answer

Units: pp

Now the interval. For each rate we use the Wilson interval. It behaves well when rates are small, like 4%, where the simple ± formula can mislead.

Alex Tutor

\frac{p + \frac{z^2}{2n} \pm z\sqrt{\frac{p(1-p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

Wilson 95% interval for a rate p = x / n, with z = 1.96. Control: 3.35% to 4.25%. trial_first: 4.58% to 5.63%.

For the difference, combine the two Wilson intervals. Each bound of the effect borrows the matching distances from both rates. Our A/B test results calculator uses the same method.

Alex Tutor

L = d - \sqrt{(p_T - L_T)^2 + (U_C - p_C)^2}, \quad U = d + \sqrt{(U_T - p_T)^2 + (p_C - L_C)^2}

Interval for d = p_T − p_C (treatment − control) from the Wilson intervals [L_T, U_T] and [L_C, U_C].

You

So for the lower bound: how far trial_first could fall, and how far control could rise.

Alex Tutor

Question 3

Compute the lower bound of the 95% interval for treatment − control, in percentage points. Use p_T = 5.08%, p_C = 3.77% and the Wilson intervals above. Two decimals.

Your answer

Units: pp

Alex Tutor

Primary: +1.31 pp, 95% interval +0.62 to +2.00 pp. The test was sized for 6,735 users per variant to catch +1 pp and got more in both. It had the power it was built for.

Guardrail, active_days_8_14: 60.53% in control, 60.60% in trial_first, +0.07 pp, interval −1.57 to +1.70 pp. The margin was −3 pp.

You

So that's not “no effect on usage”. It's “any drop is smaller than we agreed to accept”.

Alex Tutor

Question 4

Apply the decision rule from the plan: ship if the primary's interval is entirely above 0 and the guardrail's interval is above −3 pp. What is the call?

Choose one answerInconclusive: the interval includes +1 pp, so the test must be rerunShip trial_first to all usersShip on iOS onlyDon't ship: the guardrail difference isn't significant

Maya

Fine, it ships. But can I still say “trials +74%”? It sounds much better.

Alex Tutor

It's a secondary metric and a relative lift. The claim is the primary: +1.31 pp, from 3.77% to 5.08%, interval +0.62 to +2.00. Trials explain how it worked, not whether.

Theo

And the platforms? iOS is +2.00 pp, Android +0.29 pp with an interval from −0.69 to +1.28.

Alex Tutor

The test wasn't sized for platforms, and the rule is for all users. Android's interval includes 0 and +1 pp: that's inconclusive, not “no effect”. Worth a follow-up, not a separate decision.

Alex Tutor

Question 5Short answer · AI-checked task

Theo proposes shipping trial_first on iOS only, “because Android shows nothing”. Reply to him in 2–4 sentences.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Maya

I'll forward the readout to the whole team. Short, with the numbers, and something I can defend when someone asks “are you sure?”

You

And the numbers we haven't computed, like the interval for trials?

Alex Tutor

Same method, same dataset, if you want them. They won't change the decision, though.

Then write it the way the plan reads: primary with its interval, power, guardrail, the tempting numbers in their place, the decision, and what to watch next.

Alex Tutor

Question 6Case · AI-checked task

Write the readout of the paywall test for Maya.

Halves tested two Halves Plus paywalls for 4 weeks (3–30 August 2026): control (current) and trial_first (leads with a 7-day free trial). Users entered at their first paywall view, split 50/50, and were followed for 14 days. The dataset is above.

The plan, written before launch:

Primary metric: paid within 14 days (paid_14d ÷ users). Smallest effect worth shipping: +1 pp; sized for 6,735 users per variant (baseline 4.0%, α = 5%, power 80%).
Guardrail: added an expense in days 8–14 (active_days_8_14 ÷ users), margin −3 pp.
Secondary: trials started.
Rule: ship if the primary's 95% interval for treatment − control is entirely above 0 and the guardrail's interval is above −3 pp.

Write the readout for Maya (150–300 words): the effect on the primary metric with its interval, whether the test had the power it was sized for, the guardrail, what the trial and platform numbers do and don't show, the decision, and one or two things to watch after launch.

Dataset for this task: Halves paywall test: control vs trial_first

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Notice what made this easy: every question had its answer written down before the data arrived. Reading the test was just checking the numbers against the plan.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Comparing versions fairly