In a standard A/B/n test, a percentage of traffic is allocated to a “Control Group,” which receives the default experience (in a medical experiment: the placebo; in website testing: the current experience).
A “Shared Control Group” involves conducting various tests using the same control group for comparison. So, in the medical example, this would mean testing different medications against the same group ONLY receiving placebos (not as many qualified participants needed, overall!).
However, the difficulty of involving a Shared Control Group comes in deciding which other experiences to exempt your audience from (i.e., In a medical experiment, is this audience eligible to take allergy medicine? When testing a website, is this audience eligible for exposure to other experiments in unrelated site areas?).
Both the Shared Control Group and Universal Holdout are processes for withholding a percentage of eligible traffic from ALL designated exposures (marketing campaigns, site experiences, etc.) for a period. For example, this means a percentage of your audience is predetermined NOT to receive email campaigns, site testing opportunities, or app pushes during a specified timeframe.
A Universal Holdout and Shared Control Group work best when set up to complement each other. The Universal Holdout might include withholding all traffic, whereas the Shared Control Group could include a secondary protective layer that withholds traffic groups separately for different platforms, or with a more frequent refresh.
Your organization will need to agree upon a standard cadence for the refreshing of this shared control group to determine an optimal deployment setup. Refreshing in accordance with sprint cycles could be the best approach for many organizations. In other situations, or organizations, refresh intervals could depend on support around specific marketing insights.
The nature of a Universal Holdout depends on audience member identification. If a visitor is not recognized on their second visit to the site, they could become eligible for campaigns that previously deemed them ineligible (i.e., a member of the Universal Holdout may be unrecognized, and incorrectly granted exposure to active site tests).
In the case of Customer Data Platforms (CDPs), such as Tealium’s AudienceStream, piece together Customer IDs to identify otherwise unrecognizable customers by clearing cookies, cross-device visitation, or lack of authentication.
The primary goal of a Universal Holdout is to prove the effectiveness of an established marketing/testing program. By evaluating the success of the exposed (or eligible for exposure) audience against the withheld audience, a client can determine if the program—as a whole—contributes to the success of primary KPIs.
Use the option to include a Shared Control group to your advantage by implementing it in a way that works best for your testing program, audience sizes, and platform capabilities. Your experimentation initiatives might perform even better if you include a separate Universal Holdout!
Hannah Alexander and Luke Youtsey will go even deeper on this topic in their September 28, 2022 presentation at Experimentation Nation: The Conference.
For even more information about Concord’s experimentation capabilities, get in touch!
Not sure on your next step? We'd love to hear about your business challenges. No pitch. No strings attached.
