By David OlsonUnderstanding true marketing impact remains the holy grail for brands. My team recently spearheaded a campaign where we rigorously applied geo-holdout and synthetic-control incrementality testing to validate inferred credit, moving beyond last-click attribution. How did we quantify the incremental lift from our significant investment?
Key Takeaways
- Implementing a geo-holdout strategy requires meticulously defined control and test markets with similar demographic and behavioral profiles, often necessitating a minimum 10% population split for statistical significance.
- Synthetic control groups, built from a weighted combination of non-exposed regions, provide a robust counterfactual for incrementality measurement when pure geo-holdouts are unfeasible or too costly.
- Our Q3 2026 campaign achieved an incremental ROAS of 1.8x, demonstrating that 25% of conversions attributed by last-click models were actually incremental, justifying a $500,000 budget increase for similar future initiatives.
- Accurate incrementality testing demands a minimum of 4 weeks for campaign duration to allow for sufficient data collection and stabilization of market effects.
- The most significant challenge we encountered was data cleanliness and the need for advanced statistical methods to adjust for external factors affecting control groups.
I’ve seen countless marketing teams throw money at channels, pat themselves on the back for “conversions,” only to find their overall business growth flatlining. That’s because correlation is not causation, and last-click attribution is a liar. My philosophy is simple: if you can’t prove it, you didn’t do it. That’s why I’m such a staunch advocate for rigorous incrementality testing. It’s the only way to truly understand the value your marketing efforts bring.
Campaign Teardown: “Ignite Growth Q3 2026”
Our objective for the “Ignite Growth Q3 2026” campaign was ambitious: drive new customer acquisition for a B2B SaaS product, specifically targeting small to medium-sized businesses (SMBs) in the tech and professional services sectors. We allocated a total budget of $2.5 million over an 8-week period, from July 1st to August 26th, 2026. The primary performance indicators were new user sign-ups and subsequent trial-to-paid conversions.
Strategy and Targeting
Our strategy centered on a multi-channel approach: a significant portion on Google Ads (Search and Display), programmatic display via TheTradeDesk, and LinkedIn Ads. We focused our targeting geographically on 20 major US metropolitan areas, including Atlanta, Dallas, and Seattle. Within these areas, we targeted SMB decision-makers (CEOs, Marketing Directors, IT Managers) with specific job titles and company sizes (10 to 250 employees). Behavioral targeting on programmatic channels focused on users exhibiting interest in business software, cloud solutions, and productivity tools.
Creative Approach
The creative strategy leaned into problem/solution framing. For Google Search, our ad copy highlighted specific pain points our SaaS product solved, like “Streamline Project Management” or “Automate Client Onboarding.” Display and LinkedIn creatives featured short, engaging video testimonials and carousel ads showcasing key features with clear calls to action. We maintained a consistent brand message across all channels, emphasizing efficiency, scalability, and ease of use. I personally believe that creative resonance is often undervalued; even the best targeting won’t save weak creative.
The Incrementality Design: Geo-Holdout and Synthetic Control
This is where the rubber met the road. We knew we couldn’t rely on platform-reported conversions. We needed a clean read on incrementality. To achieve this, we employed a hybrid approach:
- Geo-Holdout for Tier 1 Markets: For our top 5 markets (Atlanta, Dallas, Seattle, Boston, Chicago), we implemented a true geo-holdout. We identified comparable control markets based on historical performance, demographic data (from Statista’s regional GDP data), and competitive intensity. For instance, we held out Austin, TX, as a control for Dallas, TX. In these control markets, we paused all paid media for the campaign duration. This was a tough sell internally, but absolutely essential.
- Synthetic Control for Tier 2 Markets: For the remaining 15 markets, a pure geo-holdout was either too risky (due to potential revenue loss from a complete media blackout) or difficult to implement due to market overlaps. Here, we used a synthetic control group. We identified a basket of non-exposed regions, weighting them to match the pre-campaign trend of our exposed markets. This involved meticulous data analysis, using factors like search interest, website traffic, and organic sign-ups from the preceding 12 weeks. We used a R package for synthetic control methods to construct these comparison groups. It’s a complex statistical exercise, but it offers a powerful way to create a counterfactual when a direct A/B test isn’t feasible.
Our incrementality measurement period mirrored the campaign duration: 8 weeks. We tracked key business metrics like organic sign-ups, direct traffic, and overall revenue, not just the attributed conversions from our ad platforms. This holistic view is paramount for accurate incrementality.
What Worked and What Didn’t
What Worked:
- Search Campaign Performance: Our Google Search campaigns were incredibly efficient, delivering a reported CPL (Cost Per Lead) of $75. The geo-holdout in Dallas, compared to its Austin control, showed a 15% incremental lift in organic sign-ups, demonstrating the halo effect of strong brand presence from search.
- Video Creatives on LinkedIn: The short video testimonials outperformed static images by 30% in CTR (Click-Through Rate), reaching 1.2% versus 0.9% for static. This translated to a lower cost per qualified lead on LinkedIn.
- Geo-Holdout Clarity: The true geo-holdout markets provided undeniable evidence. In Atlanta, our test market, we saw a 20% increase in new customer registrations compared to our control market, Raleigh, NC, which had flat growth during the same period. This was a clear win.
What Didn’t Work So Well:
- Broad Programmatic Targeting: Initial programmatic display targeting was too broad, leading to a high CPL of $150 and low conversion rates. We quickly iterated, narrowing our audience segments to specific B2B interest groups and implementing more aggressive negative keyword lists. This is a common pitfall; you can’t just set it and forget it.
- Attribution Mismatch: As expected, platform-reported ROAS (Return On Ad Spend) was significantly higher than our true incremental ROAS. Google Ads claimed a 3.5x ROAS, while our incrementality testing revealed it was closer to 1.8x. This discrepancy highlights why these tests are so vital.
- Data Cleanliness: We spent considerable time cleaning and normalizing data from various sources. Differences in time zone reporting, lead definitions, and integration issues between our CRM and ad platforms presented challenges. My team and I practically lived in spreadsheets for the first few weeks, ensuring data integrity. It’s a thankless job, but absolutely critical for reliable results.
Optimization Steps Taken
Mid-campaign, we made several critical adjustments:
- Programmatic Retargeting Focus: We shifted programmatic spend heavily towards retargeting website visitors and engagement with our LinkedIn content, seeing a 2x improvement in conversion rates for these segments.
- Budget Reallocation: Based on early performance, we reallocated $300,000 from underperforming programmatic display to high-performing Google Search and LinkedIn video campaigns.
- Landing Page A/B Testing: We ran continuous A/B tests on landing page headlines and call-to-action buttons, which collectively improved our conversion rate from landing page views to sign-ups by 8%.
Results and Inferred Credit
The campaign concluded with the following overall metrics (platform-reported vs. incremental):
| Metric | Platform-Reported | Incremental (Validated) |
|---|---|---|
| Total Conversions (New Sign-ups) | 15,000 | 3,750 |
| Total Cost | $2,500,000 | $2,500,000 |
| Cost Per Conversion (CPC) | $166.67 | $666.67 |
| ROAS (assuming $1,200 LTV per conversion) | 7.2x | 1.8x |
| Incremental Lift (Organic Sign-ups) | N/A | +12% in test markets |
Our analysis revealed that while platforms attributed 15,000 new sign-ups to our campaign, only 3,750 were truly incremental. This means approximately 25% of the attributed conversions would not have happened without our paid media efforts. The remaining 75% were likely organic or direct conversions that were simply “claimed” by the last-click model. This is a stark reminder that platform data should always be taken with a grain of salt. Our incremental ROAS of 1.8x, while lower than platform-reported, still represented a positive return on investment, justifying the campaign spend.
We found that the synthetic control group for our Tier 2 markets provided results that closely mirrored the directional insights from our true geo-holdouts, giving us confidence in its application for future campaigns where full holdouts are impractical. This hybrid approach is, in my opinion, the future of incrementality testing for many businesses. It balances scientific rigor with business realities.
One editorial aside: Many marketers shy away from incrementality testing because it often “disproves” the rosy pictures painted by ad platforms. But ignoring the truth doesn’t make your budget more effective. Embrace the hard numbers; they’re your best guide.
The insights gained from this campaign were invaluable. We now have a clearer understanding of our true marketing efficiency and can confidently allocate future budgets to channels and tactics that genuinely drive business growth, not just vanity metrics. This kind of testing isn’t just about validating past spend; it’s about making smarter decisions for tomorrow.
Measuring true marketing impact through rigorous incrementality testing, whether via geo-holdouts or synthetic controls, provides the undeniable evidence needed to optimize spend and confidently scale what truly works.
What is geo-holdout testing?
Geo-holdout testing involves selecting geographically distinct regions, exposing one group (test markets) to a marketing campaign while withholding it from another comparable group (control markets). By comparing the performance of the test markets against the control markets, marketers can determine the incremental impact of the campaign.
How does a synthetic control group work in marketing?
A synthetic control group is constructed by creating a weighted average of multiple unexposed regions that closely match the pre-campaign trends of an exposed test region. This “synthetic” region then serves as a counterfactual, allowing marketers to estimate what would have happened in the test region without the campaign, thereby isolating the incremental effect.
Why is incrementality testing more reliable than last-click attribution?
Last-click attribution credits 100% of a conversion to the last marketing touchpoint, ignoring all prior interactions and the possibility that the customer would have converted anyway. Incrementality testing, through methods like geo-holdouts, directly measures the additional conversions that occurred solely because of the marketing effort, providing a more accurate picture of true impact.
What are the main challenges of implementing incrementality tests?
Key challenges include finding truly comparable control and test groups, ensuring data cleanliness and accurate tracking across all channels, managing internal resistance to holding back media spend in control areas, and the statistical complexity involved in analyzing the results and accounting for external variables.
How long should an incrementality test run for?
The ideal duration for an incrementality test varies depending on the product, sales cycle, and campaign intensity, but a minimum of 4 to 6 weeks is generally recommended. This allows sufficient time for the campaign to generate an effect, for market dynamics to stabilize, and to collect enough data for statistical significance.
