Intro to Incrementality Series: Part II

In our first Intro to Incrementality piece, we explained what incrementality analysis is and how marketers can use it to make more informed decisions about what success means for their media campaigns. We mentioned how, at the very heart of this analysis, there is a comparison between two groups:

In this part of our incrementality series, we talk specifically about that second group, diving deeper into how to create a control group and what it should be composed of for unskewed analysis.

Real-World Example

Remember our basketball example? Here’s a quick refresher:

An NBA basketball team has two players who both make 40% of their foul shots. The team wants to improve that percentage, so it decides to hire a shooting coach. It designs a test to evaluate that coach and assigns him to only one of the two players. Both players are told to do everything else they had been doing, exactly as they had been doing it. For instance, they’re told to spend the same amount of time in the gym, keep a similar diet, and maintain the same weight. 

Player A: 40% of free throws ┃ Player B: 40% of free throws

After a year, the player who worked with the shooting coach makes 80% of his free throws the following season, while the player not assigned the shooting coach makes 50% of his free throws.

Player A: 50% of free throws ┃ Player B: 80% of free throws

There are some very important concepts embedded in this example but two are chief among them. First, both players make 40% of their shots. The observed behaviors of both groups are similar, if not the same. Second, everything else going on—except the shooting coach—remains the same. No after-practice shots allowed, no extra weight room, etc. Allowing additional variables into the equation creates noise that can skew or obscure incrementality analysis.

Controlling The Control

Bringing this back to marketing, we’re left with this challenge: how can we create a control group where the observed or expected behaviors match the exposed group, and how can we limit the noise around this analysis? Prior to 2020, there were three common methods for creating control (or holdout) groups for marketing and advertising campaign analysis:

1. Market A/B Testing 

Pick two similar markets, turn media or a certain type of media on in one market, leave it off in the other, and observe behavior in each. The issue with market A/B testing is that no two markets behave exactly alike, no two markets contain populations that are comprised of the exact same demographics and interests, and the background noise can reach dangerous levels.

2. Random Suppressed Groups

Using existing technology and cookie, IP, or Mobile Ad ID (MAID) lists, agencies and advertisers were able to actively hold out randomly generated subsets of people from seeing ads. However, as soon as targeting is applied to the campaign, the analysis skews heavily toward the exposed group. For example, imagine you’re selling men’s running shoes and you want to run incrementality analysis. Your campaign is targeting men, but your randomly-generated control is observing men and women equally. Your exposed group is going to perform better because half the control group can’t even buy your product.

3. PSA Ads

The exposed group, with all its relevant demographic and behavioral targeting, is served ads for the brand. Meanwhile, that same demographic and behavioral targeting is applied to a different group, and instead of ads for the brand, they’re served ads for public service or completely unrelated products or activities. The two groups remain separate and distinct but mirror each other in behaviors essentially (e.g. back to the basketball example: they both shoot 40% at the line). This allows us to see a much-less-skewed, noise-filled outcome of the incrementality test running.

The main drawback of PSA holdouts is that someone has to pay for that PSA ad to be served. Impressions don’t come free, and in order to run this analysis, impressions must actually be purchased to run PSA ads on, so a substantial portion of the advertising budget is going toward actively NOT advertising! Additionally, when a PSA is served, it actually prevents competitive brands from winning that same impression and possibly swooping in on a customer, which might be good for business but is very bad for statistics—and creates a skew toward the exposed group.

A More Sophisticated Approach

In 2020, a new method for creating a holdout emerged called ghost bidding, which allows media buyers to create a holdout by setting a cadence against individual line items or ad groups. The cadence dictates how often the buying platform will send an eligible bid into the control group, instead of actually bidding on the impression. So, at a ghost bid cadence of 20%, for every 5 impressions bid on, 1 user receives a “ghost bid”, or gets placed in a holdout group where it cannot receive ads for the duration of the campaign.

Because the ghost bid isn’t an actual impression, the ad space remains open and the “ghost bid” is free. Additionally, because it’s set against the line item or ad group itself, all the targeting matches the exposed group exactly and all optimization on the exposed group follows suit on the control group. If Millennial Urban Men are converting at a higher rate and receiving more budget in the exposed group, they’re also receiving more ghost bids, making sure the composition of the control follows suit.

Ghost bidding addresses the pitfalls of market tests, random samples, and PSAs while remaining inexpensive and relatively easy to implement. As a result, ghost bidding has become increasingly popular with sophisticated marketers and media buyers. It is Digital Remedy’s preferred method for creating holdouts for our Digital Remedy Platform.

In our next piece of this series, learn why optimizing incrementally is a crucial capability for brand marketers and agencies looking to drive tangible bottom-line results. Can’t wait? Speak to a member of our team today.