Incrementality testing: what it measures, and the four ways to do it.
Incrementality is the sales that would not have happened without the marketing. Attribution cannot measure it, because attribution only ever looks at the sales you made. Measuring it means comparing against a group that was not marketed to. There are four ways to build that comparison, each with a different cost, a different speed and a different kind of blind spot.
01
What incrementality means
What it is
Attribution gives credit: it looks at the sales you made and decides which touchpoint gets to claim each one. Incrementality asks a different question: how many of those sales would have happened anyway? A customer who searched your brand name and clicked the ad at the top was already on the way. The ad gets the credit; the sale was not incremental.
How to use it
Treat incremental sales as the only number that justifies spend. Incremental revenue is the revenue in the group that saw the marketing minus the revenue in a matched group that did not. Everything else on a platform dashboard is a claim about credit, which is useful for comparing campaigns inside one platform and useless for deciding whether the channel should exist.
02
Incrementality testing vs A/B testing
What it is
They are often confused because both split an audience. An A/B test shows two groups two versions of something, a headline, a landing page, a creative, and asks which did better. Both groups were marketed to. An incrementality test shows one group the marketing and the other group nothing, and asks whether the marketing did anything at all.
How to use it
Use an A/B test to improve a channel you have already decided to fund. Use an incrementality test to decide whether to fund it. Running A/B tests on a channel whose incremental effect has never been measured is tuning the engine of a car that may not be moving.
03
Method one: the geo holdout
What it is
Switch the channel off in some regions and leave it on in matched others, then compare what happens to total sales. It works for any channel that can be targeted by region, including ones with no click to track: out of home, radio, TV, influencer pushes, upper-funnel video. It needs enough regions and enough volume to see a gap through the week-to-week noise.
How to use it
This is the method to reach for when the budget decision is large and the channel can be geo-targeted. Match the regions on the shape of their revenue line, run a pre-period to prove they move together, and compute the smallest effect the test could detect before you start. The full six steps are in the geo holdout guide linked below.
04
Method two: the platform lift study
What it is
The large ad platforms can run a holdout at the level of individual users: they randomly keep a share of your target audience from seeing your ads and compare conversions between the two groups. The randomisation is better than any geography can offer. The drawbacks are that the platform picks the holdout, counts the conversions and reports the result, and that only the conversions reported to it, through its pixel, server events or uploaded sales, count.
How to use it
Use it for a channel that is already a large share of spend, as a check on that platform's own attributed numbers. Read the lift as a range, not a point. Then compare the lift it reports with the revenue the platform attributes to itself in the same period: the gap between the two is how much of the attributed revenue was not incremental.
05
Method three: the on/off test
What it is
Pause the channel everywhere for a period and compare sales with the period before. It is the cheapest test and the easiest to run, and it is also the easiest to misread, because anything else that changed in the same weeks, a promotion, the weather, a competitor's launch, payday, lands in the result.
How to use it
Use it when geography cannot be split and the platform offers no lift study. Remove seasonality first by comparing against the same weeks a year earlier, or against a forecast built from the pre-period, and report the result as weaker evidence than a controlled test. A before-and-after read is still better than no read, as long as nobody mistakes it for proof.
06
Method four: marketing mix modelling, calibrated by tests
What it is
Marketing mix modelling (MMM) estimates each channel's contribution statistically from weeks or years of spend and sales data, with no holdout. It covers every channel at once and never requires switching anything off. On its own it is an estimate with wide uncertainty, and with enough modelling choices it can be tuned towards the answer the analyst expected.
How to use it
Use the model for the ongoing budget view across channels, and use tests to anchor it. A geo holdout or lift study on one channel gives the model a known point to be calibrated against. The combination is stronger than either alone: tests for the truth about a few channels, the model for a consistent picture of all of them.
Which method to use
A large budget line on a channel that can be targeted by region
Geo holdout
Measures total sales, including the ones no click tracks. Independent of the platform.
One ad platform is most of your spend and you doubt its attributed revenue
Platform lift study
True random holdout at user level. Read the range, then compare it with the platform's own attribution.
A smaller channel, no geographic split possible, no lift study offered
On/off test, seasonality removed
Cheap and fast. Report it as weaker evidence than a controlled test.
You need one view across every channel, every month
MMM, calibrated by tests
Covers everything at once. Only trustworthy where a test has anchored it.
How to read the result, with illustrative numbers
Say the regions where a channel kept running booked $1,000,000 over the test, and the matched holdout regions, scaled to the same size, booked $920,000. The incremental revenue is $80,000, a lift of about 8.7% on the holdout. If the channel cost $40,000 in those regions over the same weeks, the incremental return is 2.0: two dollars of revenue the channel caused, for every dollar spent.
Now the part that decides whether the number means anything. Every test result comes with a range. If the range on that 8.7% runs from 3% to 14%, the channel works and the honest report is the range. If the range runs from minus 2% to 19%, it includes zero, and the test did not settle the question. Reporting the second case as “8.7% incremental” is the most common way a well-run test becomes a bad budget decision.
Compare the incremental return with what the platform attributed to itself in the same period. If the platform claimed $200,000 of sales on that $40,000, a ROAS of 5.0, and the test found $80,000, then 60% of the attributed revenue would have happened anyway. That ratio, incremental over attributed, is the correction factor to apply to the dashboard until the next test. Numbers changed for illustration; the arithmetic is the point.
Measure what the marketing caused, not what it claimed.
Decifer matches the regions, checks the test can detect a real effect before it starts, and reports incremental revenue as a range next to the revenue the platform attributed to itself.