Intro to Incrementality Analysis Series: Part I

Welcome to the first installment of our Intro to Incrementality series, where we’ll use real-world examples to explore how to uncover the true impact of your advertising campaigns. By isolating the effect of specific media types or campaign variables, incrementality analysis provides actionable insights into what’s driving performance–and what isn’t.

Warning: this post involves (basic) math.

Isolating the Impact

In today’s crowded media landscape, customers are bombarded with ads across multiple channels—from paid search and display to OTT/CTV and streaming audio. As a result, performance marketing has devolved into a game of attribution, focusing on whether an action occurred after an ad was served. This has largely been driven by platforms like Google and Meta, prioritizing their own attribution models to capture the lion’s share of ad dollars, and by marketers eager to show results.

While attribution can show whether an action occurred after an ad was served, it fails to answer the crucial question: Did this ad actually cause the action, or was it just along for the ride?

Enter incrementality analysis. By comparing a test group (exposed to a new campaign, publisher, creative, or other variable) with a control group (unexposed), you can measure the true lift in media performance caused by a specific variable and separate correlation from causation.

The Basketball Analogy

Let’s use a non-marketing example to simplify the concept. Imagine an NBA team with two players who each make 40% of their free throws. To improve performance, the team hires a shooting coach to work exclusively with one of the players, Player B. Both Players A and B are told to maintain all other factors, like gym time, diet, and training routines.

2024-25 Season:
Player A: 40% of free throws ┃ Player B: 40% of free throws

After the 2024-25 season, Player B’s free throw percentage improves to 80%, while Player A’s performance naturally rises to 50%. Both players improved, but Player B improved more. So, how do we measure the shooting coach’s impact?

2025-26 Season:
Player A (control): 50% of free throws ┃ Player B with Coach (test): 80% of free throws

We calculate two metrics to measure the impact:

1. Incremental Lift: After introducing the shooting coach, how much more likely is Player B to make a foul shot?

Formula:

(test: 80% – holdout: 50%) / holdout: 50%
The result is 60%

A 60% lift means Player B was 1.6 times more likely to make a free throw after working with the shooting coach.

60% more likely to make a free throw

2. Incrementality: What percentage of Player B’s improvement is attributable to the shooting coach.

Formula:

(test: 80% – holdout: 50%) / test: 80%
This result is 37.5%

Incrementality reveals that 37.5% of Player B’s improved free throw performance is directly due to the new coach, while the remaining improvement stems from other factors. 

Backing it Back to Marketing

Now, let’s apply this concept to advertising. Imagine a brand running search and social media campaigns, both yielding a conversion rate of 0.1%. The brand has a hypothesis: Adding OTT/CTV advertising to its media mix will increase the conversion rate.

To test this, the brand introduces OTT/CTV ads to one audience subset (test group) while keeping the other subset as a control (search and social). At the end of the campaign:

• The test group exposed to OTT/CTV converts at 0.2%.

• The control group (search and social only) converts at 0.15%.
  

Using incrementality analysis, we can calculate:

1. Incremental Lift: 

(test: .2% – control: .15%) / control: .15% = 33.33% (1.33x)

2. Incrementality:

(test: .2% – holdout: .15%) / test: .2% = 25%

This means OTT/CTV ads made the test group 33.33% (1.33x) more likely to convert and were directly responsible for 25% of conversions.

Incrementality analysis allows marketers to isolate and quantify the impact of specific campaign variables, leading to smarter decisions about budget allocation and media strategy. Instead of just tracking performance, you can:

• Optimize Campaigns: Reallocate budget to tactics driving true lift.
• Prove ROI: Identify which channels contribute to conversions.
• Drive Better Outcomes: Allocate your budget to ads that deliver actual incremental performance, avoiding spend on campaigns and publishers that take attribution credit for conversions and KPI results that would have happened anyway.

Looking Ahead

By measuring the true impact of advertising, incrementality analysis helps brands and agencies focus their budgets on the campaigns and publishers that matter most. In the next part of our series, we’ll discuss how to create effective control groups for accurate incrementality analysis. Can’t wait? Speak to a member of our team today.