Marketers often rely on platform-reported metrics such as conversions, click-through rates, and return on ad spend (ROAS). However, these metrics do not always reflect the true incremental impact of advertising. Studies have shown that up to 30–50% of reported conversions can occur even without ad exposure, due to organic demand or other marketing channels.
A holdout test is one of the most reliable methods to measure the real causal impact of Facebook ads. It isolates the effect of advertising by comparing a group exposed to ads with a similar group that is not.
What Is a Holdout Test?
A holdout test (also known as an incrementality test) is an experimental design where your audience is split into two groups:
-
Test group: Users who see your ads
-
Control group: Users who are intentionally excluded from seeing your ads
By comparing outcomes between these groups, you can determine how many conversions are truly driven by your campaigns.
Why Holdout Testing Matters
Relying solely on platform attribution can lead to overestimating performance. According to industry research:
-
Incrementality tests often reveal that 20–60% of conversions attributed to ads would have happened anyway
-
Brands that adopt incrementality testing reduce wasted ad spend by up to 25%
-
Proper experimentation can improve budget allocation efficiency by 15–30%
Without holdout testing, optimization decisions are based on incomplete or biased data.
Step-by-Step Guide to Running a Holdout Test
1. Define Your Objective
Clearly define what you want to measure. Common objectives include:
-
Purchases
-
Lead submissions
-
App installs
-
Subscription sign-ups
Choose a primary KPI that aligns with business goals.
2. Split Your Audience
Divide your audience randomly into two statistically similar groups. A common approach is:
-
80–90% test group
-
10–20% control group
Ensure both groups are large enough to produce statistically significant results.
3. Exclude the Control Group
Use Facebook Ads Manager to exclude the control group from your campaigns. This can be done by:
-
Creating a custom audience for the control group
-
Applying it as an exclusion in all relevant ad sets
Consistency is critical: the control group must not see any of your ads during the test period.
4. Run the Campaign Normally
Launch and optimize your campaigns as usual for the test group. Avoid making drastic changes during the experiment, as this can distort results.
The test should run long enough to capture meaningful data. Typically, this is at least 2–4 weeks, depending on conversion volume.
5. Measure the Results
Incremental lift shows the true effect of ads by comparing exposed and non-exposed audiences
Compare performance between the test and control groups. The key metric is incremental lift:
Incremental Lift (%) = (Test Conversion Rate – Control Conversion Rate) / Control Conversion Rate × 100
For example:
-
Test group conversion rate: 5%
-
Control group conversion rate: 4%
Incremental lift = (5% – 4%) / 4% = 25%
This means your ads generated a 25% increase in conversions beyond what would have happened organically.
6. Calculate Incremental ROAS
To understand financial impact, calculate incremental ROAS:
Incremental ROAS = Incremental Revenue / Ad Spend
This provides a more accurate view of campaign profitability than platform-reported ROAS.
Common Mistakes to Avoid
Small Sample Sizes
Insufficient data leads to unreliable results. Ensure your test reaches statistical significance before drawing conclusions.
Audience Contamination
If users in the control group are exposed to ads through other campaigns, results become invalid. Strict exclusion is essential.
Short Test Duration
Running tests for too short a period can produce misleading results due to natural fluctuations in demand.
Over-Optimization During Tests
Frequent changes to targeting, creatives, or budgets can skew outcomes. Keep variables stable during the experiment.
Best Practices for Accurate Results
-
Use randomized audience splits to avoid bias
-
Maintain consistent messaging and budget levels
-
Track both conversion rate and revenue metrics
-
Repeat tests periodically to validate results over time
Incrementality is not static—consumer behavior and market conditions change, so ongoing testing is necessary.
When to Use Holdout Tests
Holdout testing is especially valuable when:
-
Scaling ad budgets
-
Launching new campaigns or audiences
-
Evaluating retargeting effectiveness
-
Comparing different channels or strategies
It is less useful for very small campaigns with limited data.
Conclusion
Holdout testing is the gold standard for measuring the true impact of Facebook advertising. By isolating the effect of your campaigns, you can make more informed decisions, reduce wasted spend, and improve overall marketing efficiency.
Instead of relying solely on platform-reported metrics, adopting a rigorous testing approach ensures that your growth is driven by real, incremental results.
Recommended Reading
