Incrementality Testing: How to Measure What Your Ads Really Add
Incrementality testing is a controlled experiment that shows how many conversions your advertising caused, as opposed to conversions that would have happened anyway. You split a audience or market into a group that sees your ads and a holdout group that does not, then compare outcomes. The gap between the two is your incremental lift, and it is the closest thing paid media has to a clean answer to “did this spend work?”
This guide covers how a test works, the math with a worked example, how to choose between audience holdouts, geo tests and platform lift studies, and how to read the result without fooling yourself.
TL;DR
- An incrementality test compares a group exposed to ads with a randomly selected holdout that is not, so the difference is causal rather than correlated.
- Lift = (test conversion rate − holdout conversion rate) ÷ holdout conversion rate. Incremental conversions = that rate gap × test group size.
- Attribution reports can credit ads for sales that were coming anyway. A test tells you how much of the credit is real, and the answer is often lower than the dashboard says.
- Use audience holdouts when the platform can randomize people, geo tests when it cannot, and marketing mix models for the long-run, cross-channel view. Experiments can calibrate those models.
- Run the test long enough to cover your conversion lag, and change nothing else in the campaign while it runs.
Table of Contents
- What incrementality testing is
- How to calculate incrementality
- Incrementality testing vs. A/B testing
- Incrementality testing vs. marketing mix modeling
- Choosing a test design
- How to run a holdout test, step by step
- Reading results and avoiding common mistakes
- When to invest in incrementality testing
- Where Loop Optimizations fits
- FAQ
- Sources
What incrementality testing is
Most ad reporting is built on attribution: a person clicked or viewed an ad, later bought, and the ad gets credit. The problem is that people who see your ads are not a random slice of the population. Many were already searching for your brand or were about to buy. Crediting the ad for their purchase counts sales you would have had regardless.
Incrementality testing fixes this with randomization. Academic work on advertising measurement describes it as a randomized-control-trial procedure in which a random part of the population is prevented from being targeted by advertising, which lets researchers measure the causal effect of a campaign. The same research frames the underlying issue: after-the-fact attribution cannot isolate that causal effect without strong assumptions about unobserved factors, while a randomized test does not need them.
You will see the same idea under other names: a marketing holdout test, a ghost-ads study, a geo experiment, or a conversion lift study offered inside an ad platform. The mechanics differ, but each one withholds ads from a comparable group and measures the difference.
A simple example
Imagine an online retailer that spends $27,000 over four weeks on a paid social prospecting campaign. The platform reports 1,620 purchases, a cost per acquisition of about $16.67, which looks excellent.
To test it, the retailer randomly assigns its target audience: 90,000 people are eligible to see the ads, and 10,000 are held out. After four weeks:
| Group | People | Purchases | Conversion rate |
|---|---|---|---|
| Exposed (test) | 90,000 | 1,620 | 1.80% |
| Holdout (control) | 10,000 | 150 | 1.50% |
The holdout bought at 1.5% without seeing a single ad. So most of the “attributed” purchases would have happened anyway. The numbers below use this example throughout (they are illustrative, not a benchmark).
How to calculate incrementality
Three formulas cover most reads of a test.
- Incremental conversions = (test rate − holdout rate) × test group size. Here: (0.018 − 0.015) × 90,000 = 270.
- Lift = (test rate − holdout rate) ÷ holdout rate. Here: 0.003 ÷ 0.015 = 20%.
- Share of conversions that were incremental = incremental conversions ÷ total test conversions. Here: 270 ÷ 1,620 ≈ 16.7%.
Then translate to money:
- Incremental cost per acquisition (iCPA) = spend ÷ incremental conversions. $27,000 ÷ 270 = $100, against the $16.67 the platform reported.
- Incremental ROAS (iROAS) = incremental revenue ÷ spend. If each purchase is worth $120, incremental revenue is 270 × $120 = $32,400, so iROAS is 1.2.
That does not mean the campaign was a failure. An iROAS of 1.2 may still be worth it if customers come back and repurchase, or if the campaign feeds a funnel that other channels close. It means the decision should rest on $100 per incremental customer, not $16.67.
Watch for unequal group sizes
In the example, the exposed group is nine times larger than the holdout. That is normal: you want to withhold as few potential customers as possible. The formulas above scale the comparison correctly because they use rates, not raw counts. Raw counts would make the exposed group look nine times better.
Incrementality testing vs. A/B testing
People often ask whether this is just A/B testing. The two share a structure (randomized groups, a measured difference) but answer different questions.
- A/B testing compares variants: headline A against headline B, or landing page 1 against landing page 2. Everyone in the test sees some version of your marketing. Its job is to optimize specific elements.
- Incrementality testing compares marketing against no marketing. The control group gets no media, so the test evaluates the overall impact of the channel or campaign.
A practical way to remember it: A/B testing tells you which ad is better, incrementality testing tells you whether the ad is worth running at all. You need both. Once a lift test confirms a channel adds real sales, creative and landing-page testing improves how much it adds. Our guide to ad creative testing covers that second step.
Incrementality testing vs. marketing mix modeling
Marketing mix modeling (MMM) is the other major tool for causal-ish measurement, and the two work best together.
- An incrementality test is an experiment that measures the causal effect of a specific intervention by comparing treatment and control outcomes.
- MMM uses aggregate time-series data, such as market-level conversions and spend, to estimate how each channel responds to changes in budget.
The same ADKDD presentation notes that experiment results can be used to calibrate media mix models. In practice, that means your tests give the model ground truth, and the model gives you a continuous, always-on view between tests.
| Incrementality test | Marketing mix model | Attribution | |
|---|---|---|---|
| Method | Randomized experiment | Statistical model on aggregate data | Rules or models on tracked paths |
| Answers | Did this tactic cause sales? | How does each channel respond to spend? | Which touchpoints preceded a sale? |
| Timing | A defined test window | Ongoing, retrospective | Ongoing, near real time |
| Main weakness | Costs holdout sales; one channel at a time | Needs long history; coarse | Credits sales that would have happened anyway |
Choosing a test design
There are three common designs. Which one fits depends on what you can randomize.
Audience holdouts
The platform or your own CRM randomly withholds ads from a portion of a defined audience. This is the cleanest design when you can control exposure at the person level, for example with a customer list in an email or paid social retargeting program. It suits retargeting, CRM and lifecycle campaigns, where the audience is known.
Platform conversion lift studies
Major ad platforms offer built-in lift studies that handle the randomization and the statistics for you. They are convenient and usually require less setup, but they measure only that platform’s ads, usually under that platform’s own rules for eligibility, conversion volume and duration. Check the current requirements in the platform’s help center before planning, because they change. Use them when you want a fast read on one channel.
Geo experiments
When you cannot randomize individuals (TV, out-of-home, some search and video, or any case where tracking is limited), you randomize places. You pick comparable regions, turn ads off in some and keep them on in others, then compare sales. Geo tests do not depend on cookies or user-level identifiers, which makes them durable as tracking becomes harder. They need enough regions with similar, stable sales patterns, and results are noisier than person-level tests. The ADKDD geo-experiment work cited above is a good starting point for the statistics.
Quick selection guide
| Situation | Best fit |
|---|---|
| You have a known audience and can suppress ads to part of it | Audience holdout |
| You want a fast read on one platform and meet its volume requirements | Platform lift study |
| Offline or untrackable channels, or privacy-limited measurement | Geo experiment |
| You want to compare many channels over time | MMM calibrated by tests |
How to run a holdout test, step by step
- Write the question. “Does our prospecting campaign on paid social drive purchases we would not otherwise get?” is testable. “Is our marketing working?” is not.
- Pick one primary metric. Purchases, revenue or qualified leads, measured the same way for both groups. Decide this before you see data.
- Size the holdout. Smaller holdouts cost fewer sales but take longer to reach a clear result. A common range is 10% to 20%, but the right size depends on your volume and the lift you expect. If your conversion volume is low, you may not be able to detect a modest lift at all, and that is worth knowing before you spend.
- Randomize properly. Assign at the person (or region) level before anyone sees an ad. Do not build a holdout from people who “did not click”, since that reintroduces the bias you are trying to remove.
- Freeze the variables. Avoid big budget changes, new creative launches or promotions in the middle of the test, or at least apply them equally to both groups.
- Run past your conversion lag. If customers typically take two weeks to buy after first contact, a one-week test will understate lift. Include a post-exposure window.
- Read the result with a confidence interval. A 20% lift with a range of 2% to 38% is a very different finding than 20% with a range of 17% to 23%.
- Act and retest. Shift budget, then retest the changed setup. Incrementality drifts as audiences saturate and competitors move.
Clean data underpins every step. If purchases are missing from your tracking in one group more than the other, your lift number will be wrong. Server-side conversion tracking helps keep conversion signals consistent when browsers block client-side tags.
Reading results and avoiding common mistakes
Do not over-read a single test. A result describes one audience, one period and one creative set. A strong holiday-season lift does not guarantee the same in February.
Check statistical significance and power. An underpowered test returns a wide interval that includes zero. The correct reading is “we cannot tell,” not “the channel does nothing.”
Beware contamination. If the holdout group sees your ads through another campaign or channel, the gap shrinks and you will undercount lift. Exclude holdouts across every campaign that targets the same people.
Match the metric to the business. Lift in cheap, low-value conversions (free sign-ups) can hide the absence of lift in paid customers. Where possible, measure revenue or qualified pipeline. If you optimize bids toward real value, value-based bidding pairs well with test results.
Remember what a test cannot do. It measures the tested tactic during the test window. It does not capture every long-term brand effect, and it costs you some sales in the holdout. Those limits are the reason many teams combine experiments with an ongoing model.
Do not treat low incrementality as automatic failure. Branded search and retargeting often show low incremental share because those buyers were already close to purchasing. That is a signal to trim spend there and put it toward prospecting, not necessarily to cancel the channel.
When to invest in incrementality testing
Testing costs time, some foregone sales and analyst attention, so it pays off most in specific situations:
- Spend on a channel is large enough that a 10% to 20% misallocation would be material.
- Platform-reported ROAS looks too good, or the channels’ reported conversions add up to more than your actual sales.
- You are about to scale a channel and want to know whether returns will hold.
- Your tracking has degraded and attribution is less reliable than it was.
If you spend a few hundred dollars a month, a large formal test is not worth it. Start with simpler discipline: consistent tracking, creative testing and landing-page alignment.
Where Loop Optimizations fits
A lift test tells you whether a channel adds sales. The next question is how to get more from the spend that proves out. Loop Optimizations continuously analyzes your website, launches and tests multiple ad variants across Google, Meta, TikTok and LinkedIn, and rewrites landing pages to match each ad’s promise, then reports statistically significant winners. Our methodology page explains how the loop works.
Pricing is published: Website Optimization is $497 per month, each connected channel adds $297 per month, and the Agency plan is $499 per month. See the details on the pricing page.
If you run paid campaigns and want ad and page improvements to compound while you validate channels with lift tests, explore Loop Optimizations.
FAQ
What is an example of an incrementality test?
A retailer randomly withholds a paid social campaign from 10% of its target audience for four weeks. If the exposed group converts at 1.8% and the holdout at 1.5%, the lift is 20% and only part of the reported conversions were truly caused by the ads. The worked example above walks through the full calculation.
How long should an incrementality test run?
Long enough to cover your typical time from first ad exposure to purchase, plus enough volume to reach a clear confidence interval. For products with a short buying cycle, that often means a few weeks; for longer sales cycles it can mean more. Define the duration before launch and avoid stopping early when numbers look good.
Is a conversion lift study the same as a holdout test?
They use the same logic. A conversion lift study is usually the version run inside an ad platform, which manages randomization and reporting for you. A holdout test is the general term and can be run by you, with your own audience lists or regions.
Can small businesses do incrementality testing?
Yes, but volume is the constraint. If you get only a handful of conversions per week, a test may not detect a realistic lift. In that case, run longer tests on your largest channel, use geo splits if you have regional sales, or focus first on creative and landing-page testing.
Should I replace attribution with incrementality testing?
No. Attribution gives you daily, granular reporting. Incrementality testing gives you periodic, trustworthy calibration. Use tests to check how much to trust your attribution and where to move budget.
Sources
- A Large Scale Benchmark for Uplift Modeling (ADKDD 2018)
- Advertising Incrementality Measurement using Controlled Geo-Experiments (ADKDD 2020)
- Incrementality testing for marketers (Supermetrics)