Key takeaways
- Incrementality is the part of a result your marketing caused. You find it by comparing a group that got the marketing with a matched group that was held back.
- Incremental lift is (test - control) / control. Judge spend on incremental CPA and iROAS, because platform CPA also counts conversions that would have happened anyway.
- At Shopify, geo holdouts I ran with Data Science found $2M+ of paid social spend that wasn't adding merchants. The budget moved to channels that were.
- Hold out a slice of anything you can't otherwise prove. At Commas that meant a 15% holdout on a 20,572-account upgrade test and a holdout on every lifecycle flow.
01What incrementality means
Incrementality in marketing is the share of a result that happened because of a campaign and would not have happened without it. You measure it by comparing a group that saw the marketing with a matched group that was held back, and the gap between them is the incremental lift.
The definition matters because dashboards count something else. An ad platform takes credit for every conversion it touched, and some of those buyers were already on their way. Someone searches your brand, clicks a retargeting ad and checks out. The ad was there. It may have caused nothing.
An incrementality test, also called a holdout or lift test, answers the question that should move budget: if I turned this off, what would I lose?
02How to calculate incrementality
The basic incrementality formula is lift. The other 4 turn it into numbers you can budget with. A rate is conversions divided by the people, or regions, in each group.
| Metric | Formula | What it tells you |
|---|---|---|
| Incremental lift | (test rate - control rate) / control rate | How much the marketing raised the outcome, in percent |
| Incremental conversions | (test rate - control rate) × people in the test group | How many conversions the spend caused |
| Incremental CPA (iCPA) | spend / incremental conversions | What each caused conversion really cost |
| Incremental ROAS (iROAS) | incremental revenue / spend | Revenue caused per dollar spent |
| Incrementality factor | incremental conversions / platform-reported conversions | How far to discount platform numbers until the next test |
A worked example
These are illustrative round numbers, not results from a real test. You hold out 10% of an audience of 1,000,000 and run for 4 weeks.
| Illustrative | Test group | Holdout |
|---|---|---|
| People | 900,000 | 100,000 |
| Conversion rate | 1.2% | 1.0% |
| Conversions | 10,800 | 1,000 |
| Spend | $180,000 | $0 |
| Conversions the platform claimed | 7,200 | None |
Lift is (1.2 - 1.0) / 1.0, or 20%. Incremental conversions are 0.2% of 900,000, or 1,800. The other 9,000 would have happened anyway. Incremental CPA is $180,000 / 1,800 = $100, while the platform's 7,200 claimed conversions implied a $25 CPA, 4x too cheap. The incrementality factor is 1,800 / 7,200 = 25%. At $250 of revenue per conversion, iROAS is 2.5. The platform would have reported a ROAS of 10.
03Incrementality vs attribution
Attribution splits credit for conversions that happened across the touchpoints before them. Incrementality tests whether those conversions needed the touchpoints at all. I use both, for different jobs.
| Attribution | Incrementality | |
|---|---|---|
| Question it answers | Which touchpoints came before each conversion | Which conversions the marketing caused |
| Method | Rules or models applied to tracked touchpoints | A controlled comparison of a test group and a holdout |
| Output | Credit by channel, campaign and ad | Lift, incremental conversions, iCPA and iROAS |
| Cadence | Always on | Periodic tests, usually weeks long |
| Blind spot | Credits conversions that would have happened anyway and misses what it can't track | A snapshot in time that needs volume and a clean control |
| Best for | Daily optimization inside a channel | Budget calls between channels and calibrating attribution |
Attribution also over-credits the channels closest to the sale and misses influence that never leaves a tracked click. At Pipe17, 32% of 2026 new-business wins came through AI assistants, revenue the CRM had credited to paid search. Asking buyers what they used surfaced it, a method I cover in the AEO playbook.
04Types of incrementality tests
There are 6 common ways to measure incrementality. Choose by what you can randomize: users give the cleanest read, regions come next and time periods come last.
| Test | How it works | Use it when | Watch for |
|---|---|---|---|
| Geo holdout | Turn a channel off in some regions while the rest keep running, then compare. | You can't randomize users, or need the whole channel effect, offline sales included. | Regional quirks and buyers who see ads in one region and buy in another. |
| Matched-market test | Pair test and control markets whose history moves together, often with a synthetic control. | You have few markets or are launching somewhere new. | A local event that breaks the match. |
| Audience or user holdout | Randomly hold back a share of users from an email, offer, flow or retargeting list. | You control delivery: CRM, lifecycle and offers. | Held-out users getting the message another way. |
| Platform conversion lift study | Meta and Google randomly hold back people. Google also offers a regional version and reports iCPA and iROAS. | You want a fast read on one platform. | The platform grades its own test. |
| PSA or ghost ads | Controls see an unrelated public service ad, or their usual ads while the system logs where yours would have run. | Display and programmatic buys. | PSAs cost money and can bias results on platforms that optimize delivery. |
| On/off test | Turn a channel off for a period and compare with before. | Nothing else is possible. | Seasonality and every other change. |
05How I ran geo holdouts at Shopify
At Shopify I ran paid merchant acquisition, $20M+ a year across 6 channels. The platforms reported plenty of conversions. The harder question was which dollars produced merchants who paid and stayed.
With Data Science I ran geo-holdout tests across paid social. They found $2M+ in spend that wasn't adding merchants, and the budget moved to channels that were.
I carry 3 habits from that work into every test I design:
- Measure the merchant who pays. When I moved bidding and reporting to paid conversions on the $1-a-month trial, signup CPA rose about 20% and cost per paying merchant fell about 30%. The two moved in opposite directions, so the metric you pick can flip the verdict.
- Read outcomes that last. The dashboard I built for budget decisions joined ad data to merchant GMV at 30, 60, 90 and 180 days by campaign. Lift that fades by day 90 isn't lift I'd pay for.
- Isolate one variable. I ran 40+ structured experiments a quarter, each designed to change one thing so the result meant something.
06Holdouts on offers and lifecycle at Commas
At Commas I designed a five-offer upgrade experiment across 20,572 never-paid accounts, with a 15% holdout and paid retention read at day 45. The holdout, about 3,100 accounts, gets no offer. It's the baseline every offer has to beat.
The holdout shows whether an offer creates paying accounts or only pulls forward payments that were coming anyway. Reading paid retention at day 45, instead of the first payment, catches offers that buy one payment and then lose the customer.
Lifecycle gets the same rule. Each of the 18 HubSpot workflows and 64 emails I rebuilt across onboarding, activation, win-back and financing is measured against a holdout. Email is the easiest place to fool yourself, because the people who open a win-back email are often the ones who would have come back on their own.
07How to design a holdout test
- Write the decision first.Say what you'll do if lift comes back at zero and what you'll do at 30%. If no result would change the budget, skip the test.
- Pick the metric that pays.Paying customers, first payments or revenue. Signups can show lift that never reaches revenue.
- Size it before you start.Take the baseline rate and its weekly noise, pick the smallest lift worth acting on, then solve for holdout size and duration. Halving that lift roughly quadruples the sample you need. If the answer is 6 months and half the country, test a bigger change.
- Choose markets on history.Many small regions beat a few big ones. Match test and control on pre-period trends as well as size, and drop regions with one-off events. Meta's open-source GeoLift package builds power analysis and market selection into one workflow.
- Block contamination.Freeze promos, pricing and creative launches in test regions. Expect spillover: people who see ads in a test region and buy in a control region shrink the measured lift. Suppress audience holdouts in every channel that carries the same message.
- Run a full cycle, then read once.Cover at least one full buying cycle and keep reading after spend stops for late conversions. Stopping early because the chart looks good inflates false positives. Google's geo lift documentation also recommends waiting for the end of the study.
08Mistakes that break a test
- Reading a small test as proof of no effect. An underpowered test that shows no lift means you can't tell yet.
- Stacking tests. Running 2 holdouts in the same regions or audience at once muddies both reads.
- Treating one result as permanent. Incrementality moves with spend level, creative, season and competition. Retest when any of them shift.
- Taking a platform's lift study as final. Check it against your own revenue data before you move budget.
09When not to bother
A holdout costs less than certainty you don't have. It still costs something: the sales you give up in the control group, weeks on the calendar and the team's attention. I skip it when:
- No result would change the decision.
- Volume is too low to detect a lift worth acting on within a quarter.
- The deal-level evidence is already lopsided. At Pipe17, deal-level analysis showed a six-figure LinkedIn line had produced no closed revenue, and the budget moved to what did.
- The campaign is too short to hold out cleanly, like a one-week launch.
Everything else that carries a real share of budget earns a holdout, and another one when it changes.
FAQQuestions people ask
- What does incrementality mean?
- Incrementality is the share of a result your marketing caused, measured against what would have happened without it. If 1,000 customers would have bought anyway and a campaign brought in 200 more, those 200 are incremental.
- What is the difference between attribution and incrementality?
- Attribution splits credit for conversions across the touchpoints that came before them. Incrementality measures how many of those conversions the marketing caused, by comparing a group that saw it with a group that was held back.
- How do you calculate incrementality?
- Incremental lift is the test group's conversion rate minus the control group's rate, divided by the control group's rate. Divide spend by incremental conversions for incremental CPA, and divide incremental revenue by spend for iROAS.
- How does incrementality testing work?
- You split people or regions into a test group that gets the marketing and a control group that doesn't, run both for a set period and compare outcomes. The difference between the groups is the lift the marketing caused.
- What is a geo holdout test?
- A geo holdout turns a channel off in a set of regions while it keeps running in matched regions, then compares results. It works when you can't randomize individual users, and it's how Farhan Munshi and Shopify's Data Science team found $2M+ of paid social spend that wasn't adding merchants.
