Retail media dashboards are good at showing sales next to ads. They are less good at showing which of those sales would have happened anyway. A shopper who searches for your brand by name and buys after clicking a sponsored listing may have bought regardless, and a report that credits the ad for that sale looks impressive without telling you whether the spend created anything new.
In short: incrementality testing measures the extra sales caused by advertising by comparing people or places who saw the ads with similar ones who did not. In retail media the main methods are holdout tests, geo experiments, matched-market comparisons and analysis inside clean rooms. All of them need a clear question, a fair comparison group and honest reading of results, including accepting when a test is inconclusive.
1Attribution Versus Incrementality
Attribution assigns credit for a sale to an ad that the shopper saw or clicked, usually within a set window. It answers the question "which ads were in the path to purchase?" Incrementality asks a different question: "what would have happened if we had not run the ads?" The gap between the two is where wasted spend hides.
The gap tends to be widest for audiences already close to buying: loyal customers, branded search terms, and shoppers on the product page. It tends to be narrower for new customers and for categories where awareness is the barrier. That is why one account can have high attributed returns and still show a modest lift when tested. For background on how networks report sales, see our guide to retail media measurement and clean rooms.
Incrementality is not a fashionable word for a cleverer attribution model. It is an experimental idea: create a fair comparison, then measure the difference. That is why the design matters more than the dashboard, and why a modest, well-run test usually teaches more than a large model with weak assumptions.
2Holdout Tests
A holdout test splits a defined audience into two groups at random. One group is eligible to see your ads. The other, the holdout, is deliberately kept from seeing them. After the campaign, you compare purchase behavior between the groups. If the exposed group buys more, the difference is the lift attributable to the advertising.
Random assignment is what makes the test credible, because it spreads differences in shopper behavior evenly across both groups. Where a retailer or platform supports audience-level holdouts, this is often the cleanest option. Where it does not, you may be limited to indirect designs, since many networks do not let advertisers control who is suppressed.
Two practical points matter. First, the holdout group must be truly unexposed, so check that the same shoppers are not being reached through another campaign or channel. Second, the group needs to be large enough to detect a difference. A very small holdout may produce noise that looks like a result.
3Geo Experiments and Matched Markets
When you cannot randomize individual shoppers, you can randomize places. In a geo experiment you run ads in some regions or store catchments and withhold them in comparable ones, then compare sales. This works well for in-store and digital out-of-home placements, regional retailers, and any setup where ad delivery can be controlled by location.
The critical step is choosing comparable regions. Markets differ in size, seasonality, competition and store mix, so teams usually pair regions with similar historical sales patterns and check that they tracked each other before the test. A pre-period analysis, looking at how both groups behaved before the ads started, helps confirm the match.
Geo tests have limits. Fewer units means less statistical power, and spillover can blur results when shoppers cross region lines or buy online from anywhere. They are best treated as a way to estimate the overall effect of a channel, rather than to optimize individual keywords or placements.
4Where Clean Rooms Fit
A data clean room is a controlled environment where an advertiser and a data owner, such as a retailer or platform, can analyze combined data without either side handing over raw records. Amazon Marketing Cloud is a well-known example. Clean rooms are not a test design in themselves. They are the place where tests and analyses are run when the data cannot leave its owner.
Typical uses include building audiences by purchase behavior, analyzing paths across ad types, measuring overlap between campaigns, and comparing exposed and unexposed groups using the retailer's transaction data. The advantage is depth: you can ask questions the standard dashboard cannot answer. The cost is effort, since queries need to be written, privacy rules respected and outputs validated. Our clean rooms and attribution service describes how that work is typically organized.
Even with a clean room, the same rules apply. The comparison group must resemble the exposed group, the time window must be fixed in advance, and the definition of a purchase must match on both sides. A clever query on an unfair comparison is still an unfair result, so involve someone who can review the design before the numbers come back.
5Comparing the Methods
| Method | How the comparison works | Best for | Main limitation |
|---|---|---|---|
| Audience holdout | Random exposed group versus random suppressed group. | Onsite and offsite campaigns where the platform supports suppression. | Depends on platform controls and on keeping the holdout clean. |
| Geo experiment | Ads on in some regions, off in similar ones. | In-store, digital out-of-home and regional retail. | Fewer units, so smaller effects are hard to detect. |
| Matched-market or synthetic control | Statistical model builds a comparison from similar markets or history. | Cases where randomization is not possible. | Relies on assumptions that need to be checked. |
| Clean room analysis | Query combined data to compare exposed and unexposed shoppers. | Deeper questions on audiences and paths. | Needs skilled setup, and the comparison is only as fair as the design. |
| Pause or on-off test | Stop spend for a period and watch sales. | A rough first read on a single tactic. | Confounded by seasonality, promotions and stock changes. |
No single method is the right one for every campaign. Many teams combine them: a holdout where the platform allows it, a geo test for in-store and out-of-home activity, and clean room analysis to understand why a result came out as it did. Where several methods point the same way, confidence rises. Where they disagree, that is a prompt to look closer rather than to pick the answer you like.
6Designing a Test That Can Be Trusted
Good tests are decided before the money is spent. Write down the question, the metric, the groups, the duration and what result would change your decision. If you only decide how to interpret the numbers afterward, it is easy to find a story in whatever came back.
- Pick one question. For example, does branded search spend add sales, or does a new offsite audience reach customers we do not already have?
- Choose a primary metric. Incremental sales, new-to-brand customers or incremental units, not all three at once.
- Keep other things steady. Avoid launching promotions, price changes or major creative swaps mid-test where you can. If something changes, record it.
- Allow enough time. Purchase cycles differ by category, so the test needs to run long enough to capture buying, not only browsing.
- Check stock and pricing. An out-of-stock product in the test group can look like an ad failure.
- Plan for power. Ask an analyst whether the audience or number of regions can reveal the size of effect you care about.
7Reading the Results Honestly
A result usually arrives as a lift estimate with a range of uncertainty. Read both. A positive estimate with a wide range may mean there is an effect, or may mean the test was too small to tell. Report it that way. Treating a noisy result as a proof will cost you later when the next test disagrees.
Consider what the test covers. A lift measured on one product at one retailer in one season does not automatically apply to everything else. Branded and non-branded terms, new and existing customers, and onsite and offsite activity often behave differently, so segment results where the design allows.
Be equally careful with halo effects. Ads can influence purchases of other products, other retailers or later periods, and a test confined to one retailer may miss some of that. State what was and was not measured. Use the findings to adjust budgets, bids and audience choices, and treat them as one input alongside attribution reports, not a replacement.
Finally, share findings in plain terms. A finance or sales colleague does not need the statistical detail, but they do need to know what was tested, what was learned and what will change. A short summary with the decision attached helps the next budget conversation, and builds trust that the numbers are being read with care.
8Making Testing a Habit
One test is useful. A regular program is better. Keep a simple log of every test: the question, the design, the dates, what was found and what you changed. Over time that becomes a record of what actually works for your brand, and a way to avoid repeating old arguments.
Start small. Pick the biggest line of spend you are least sure about, design a clean test around it, and act on the answer. If you manage several networks, our guide on managing multiple retail media networks covers how to keep reporting consistent, and our retail media marketing team can help set up tests and read the results with you. You can also contact AdOps Media to talk through a first test.
