Price increase or paywall change? Reading a price test

Maya

Two proposals for next quarter. Theo wants Halves Plus prices up 20%. The design team wants to show the paywall at a different moment. We have room for one. Which?

Alex Tutor

Let's see what evidence each one has. For price we have a real test: in June new paywall viewers were randomly shown either today's prices or prices 20% higher.

The plan, written before launch: primary metric, first-payment revenue per paywall viewer. Guardrails: conversion to paid and trial starts. Decision rule: raise prices only if the whole 95% interval of the difference is above zero.

You

Revenue per viewer, not conversion, because a higher price is supposed to lower conversion a bit.

Alex Tutor

Dataset · 2 rows

Halves Plus price test, June 2026

Randomized by user; read on 5 July, when every trial had ended.

Download CSV

Columns
arm: control (current prices) or higher_price
monthly_price: Monthly plan price shown, USD
yearly_price: Yearly plan price shown, USD (paid upfront)
paywall_viewers: New paywall viewers randomized into the arm, 1–21 June 2026
trials_started: Viewers who started the 7-day trial
paid_monthly: Viewers whose first payment was a monthly plan (by 5 July)
paid_yearly: Viewers whose first payment was a yearly plan (by 5 July)

Data

arm	monthly_price	yearly_price	paywall_viewers	trials_started	paid_monthly	paid_yearly
control	4.99	39.99	3012	548	187	74
higher_price	5.99	47.99	2988	509	166	62

Alex Tutor

Question 1

What was the primary metric, first-payment revenue per paywall viewer, in the higher_price arm? Round to cents.

Your answer

Units: USD, e.g. 1.25

Alex Tutor

Control earned $1.2923 per viewer, the higher price $1.3286. The effect, higher price minus control, is +$0.036 per viewer, +2.8%. Its 95% interval runs from −$0.30 to +$0.37.

You

It's positive, though. Could we call it almost significant?

Alex Tutor

No such thing. The interval holds a 23% loss and a 29% gain. Yearly plans arrive as $40 lumps, so revenue per viewer is noisy, and 3,000 viewers per arm can't tell a 3% effect from zero.

So the decision rule isn't met. And "not shown" isn't "no effect": the test simply lacked the power. Confidence intervals covers the maths.

Alex Tutor

Question 2

Effect on revenue per viewer (higher price − control): +$0.036, 95% interval −$0.30 to +$0.37. The rule was to raise prices only if the whole interval is above zero. Which reading is right?

Choose one answerThe price rise worked: revenue per viewer went up 2.8%The result is almost significant, so a longer run will confirm the gainThe test proves the price change has no effect on revenueThe rule isn't met: the data fits anything from a large loss to a large gain, so this test can't support the price rise

Theo

Conversion to paid fell from 8.67% to 7.63%. That's the price doing its job, right?

Alex Tutor

Up to a point. With every price 20% higher, revenue per viewer holds as long as conversion stays above control's rate divided by 1.2. That's the break-even conversion.

You

That assumes the same share of yearly plans in both arms.

Alex Tutor

Good catch. It was close: 28% of control's payers chose yearly, 27% of the higher-price arm's. Work out the break-even rate.

Alex Tutor

Question 3

Control converted 261 of 3,012 viewers to paid. With every price 20% higher and the same plan mix, what is the break-even conversion to paid, below which revenue per viewer falls?

Your answer

Units: %, e.g. 12.5

Alex Tutor

7.63% is above the 7.22% break-even, but the conversion gap's interval, −2.42 to +0.35 points, reaches below it. And the primary metric stops at the first payment.

You

The first monthly renewals at $5.99 only come in July. If more of those people cancel, the lifetime gain shrinks.

Maya

And the paywall idea? Since the redesign, paywall-to-trial went from 13.4% to 16.8%. That's a quarter more trials.

Alex Tutor

That's a before-and-after during trip season. Within every trigger the rate stayed flat or fell; the currency trigger grew from 12% to 39% of viewers. At April's mix, June converts at 12.7%.

Alex Tutor

Question 4

Which statements about the evidence are correct? Select all that apply.

Choose all that applyThe paywall redesign's 13.4% → 16.8% comes from a before-and-after comparison, not a testAt April's trigger mix, June's paywall-to-trial rate is below April'sThe price test shows monthly subscribers renew as often at $5.99 as at $4.99The price test was randomized, so its difference can be read as an effect of priceThe paywall's rise in trial starts proves the redesign works during trip season

You

So neither idea has proof. Price has a weak test, the paywall has none.

Alex Tutor

Then part of the answer is how to get proof. Before we write to Maya, sketch the paywall test: who's in it, one primary metric, and the rule you'd decide by.

Theo

Keep it short. I'd like my price rise back on the table by autumn.

Alex Tutor

Question 5Short answer · AI-checked task

Sketch an A/B test for the paywall change in 3–5 sentences: who is enrolled, the one primary metric, guardrails, and the decision rule.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Alex Tutor

Now the memo. Maya wants a choice for the quarter, the numbers behind it, what we don't know, and the next step.

Maya

One page, please. I'll forward it to the CEO as it is.

Alex Tutor

Question 6Case · AI-checked task

Write Maya a recommendation: raise the Halves Plus price, change the paywall, or neither yet?

Halves can take on one of two monetization projects next quarter:

Raise Halves Plus prices by 20% (monthly $4.99 → $5.99, yearly $39.99 → $47.99). Evidence: the June price test above. Primary metric, declared up front: first-payment revenue per paywall viewer. Rule: raise prices only if the whole 95% interval of the difference is above zero.
- Revenue per viewer: control $1.2923, higher price $1.3286; difference +$0.0363 (95% interval −$0.2979 to +$0.3705).
- Conversion to paid: 8.67% vs 7.63%; difference −1.03 points (95% interval −2.42 to +0.35). Break-even conversion for +20% prices: 7.22%.
- Renewals at the new price: not yet observed.
Change the paywall (the redesign shipped in May). Evidence: paywall-to-trial rose from 13.39% in April to 16.79% in June. Within each trigger the rate was flat or lower; the currency trigger grew from 11.79% to 39.35% of viewers (trip season). June's per-trigger rates at April's mix give 12.71%.

Maya will forward your memo to the CEO. Write 200–350 words:

Your recommendation for the quarter.
What each piece of evidence shows, with the numbers, and what it can't show.
What is still unknown and how you'd find out (including the primary metric and decision rule of any new test).

Dataset for this task: Halves Plus price test, June 2026

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Pricing decisions are rare and expensive to undo. Read the effect with its interval, name what the test couldn't see, and never let a before-and-after chart stand in for a test.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Forecasting active users