Skip to content
Log in
← Statistics glossary

incrementality

Also called: incremental lift, incrementality testing, incrementality test, lift test

The part of a result that happened only because of a campaign or channel, measured against a holdout or a geo test. Attributed conversions are often not incremental.

Incrementality asks a causal question: how many of these users or purchases would not have happened without the campaign? Attribution tells you which channel was credited; incrementality tells you what the channel actually added. The gap can be large: brand search ads, retargeting of users who were about to return anyway and reminder pushes often get credit for conversions they didn't cause.

To measure it you need a comparison group that didn't get the campaign:

  • User-level holdout. Randomly keep a share of the audience (say 10%) from seeing the ads or receiving the messages. Works for your own channels (push, email, in-app prompts) and for ad platforms that offer lift studies.
  • Geo test. Turn the campaign on in some regions and off (or down) in comparable ones, then compare. It needs no user-level data, which makes it the workhorse for paid media under today's privacy rules. Choose regions that behaved alike before the test and adjust for their pre-test difference.
  • Time-based on/off. Pause a channel for a period and watch the total. It's the cheapest option and the weakest one, because seasonality and other changes happen at the same time.

The core arithmetic:

incremental conversions=conversions in test−expected conversions without the campaign\text{incremental conversions} = \text{conversions in test} - \text{expected conversions without the campaign}

incremental CAC=spendincremental conversions\text{incremental CAC} = \frac{\text{spend}}{\text{incremental conversions}}

Compare incremental CAC, not attributed CAC, with LTV when you decide the budget. Treat the result as an estimate with uncertainty: geo tests have few units (regions), so run them long enough and report a range, not just a point estimate.

Example

Theo runs a 4-week geo test on paid social. Ads stay on in 10 test regions and are switched off in 10 matched control regions. Before the test, the test regions had 1.04 times as many signups as the control regions.

Signups in 4 weeks
Test regions (ads on)
5,200
Control regions (ads off)
4,000
Expected in test regions without ads
4,000 × 1.04 = 4,160
Incremental signups
5,200 − 4,160 = 1,040

Spend in the test regions was $10,400, and the store-framework reports attributed 2,600 signups there to paid social.

  • Attributed cost per signup: $10,400 ÷ 2,600 = $4.00
  • Incremental cost per signup: $10,400 ÷ 1,040 = $10.00
  • Share of attributed signups that are incremental: 1,040 ÷ 2,600 = 40%

Paid social is 2.5 times more expensive than attribution suggested. If a new Halves user is worth about $3.50 in gross profit over 12 months, the channel loses money at current spend, and Theo should test a lower budget or different audiences before scaling it.

Common mistakes

  • Assuming attributed = caused. Retargeting and brand search often take credit for users who were coming anyway.
  • Ignoring pre-test differences between regions. Adjust for how test and control compared before the campaign.
  • Tests that are too short or too small. A few regions over one week can't separate the effect from noise; report an interval.
  • Running the test during a season change or a store feature. Outside events hit test and control unevenly; avoid them or extend the test.
  • Measuring once and forgetting. Incrementality changes with spend level and audience; retest when the budget or targeting shifts.