Reserve a control group and measure campaign lift

Measure causation

Observed conversion answers “what happened?” A holdout experiment answers the harder question: “what happened because of this campaign?” CAB withholds a stable share of genuinely new members and compares their conversion rate with the contacted group.

Scheduled or one-time campaign action · Deterministic assignment · Conversion outcomes

Start an experiment

1. Configure a scheduled add-to-campaign action with at least one compatible conversion outcome.
2. Enter a percentage in Reserve a control group (%). CAB previews the effect as ~… of … people never contacted.
3. Save the schedule. The same people remain held out across recurring runs; new joiners are assigned when they first appear.
4. Open Outcomes or the run’s outcome section to review Lift vs control and Holdout lift.

The lift view separates Treated and Holdout, shows both sample sizes, and labels the result significant, not significant, insufficient sample, or partial — still measuring.

Inspect or conclude the experiment

Choose View / export control group to open Control group (holdout members). When the experiment should end, use Release control group. After confirmation, held-out members are added on the next run and lift measurement stops; this cannot be undone.

Reserve a control group for one manual add

The ordinary Add to Campaign flow also shows Reserve a control group (%) when Outcome Tracking is available. Enter a percentage before confirming the action. CAB partitions that one activation and shows … held out this run in its run-history outcome section.

A manual holdout is a one-time partition, not a standing experiment. It has no recurring schedule on which to release the members later, and the assignment is independent from another manual add. Use the scheduled workflow when you need the same people to remain in the control arm across repeated sends and want Release control group to add them later.

Experiment guardrails

• Only conversion-event outcomes can measure people who never became campaign members.
• Only one scheduled action may target a campaign while its control group is active.
• The percentage locks after the experiment starts to protect validity.
• A control group excludes people from this campaign only; other activity can still affect the comparison.
• Small samples do not receive an overconfident lift claim.

Causal result:

Unlike the organization-average comparisons elsewhere in Outcomes, the treated-versus-holdout result is designed to estimate campaign lift.

For admins

There is no separate holdout switch in CAB Settings. The feature uses the Outcome setup described in the previous section: membership-change tracking, the nightly outcome measurement job, and a compatible conversion definition.