Reserve a control group and measure campaign lift

A conversion rate describes what happened after activation. A control group helps estimate how much difference the activation made: CAB reserves part of the eligible audience and compares its outcome with the group added to the campaign.

Start an experiment

1. Configure a scheduled add-to-campaign action with at least one compatible conversion-event outcome.
2. Set Reserve a control group (%). The preview estimates how many members will be held back from this campaign.
3. Save the schedule. The assignment stays stable across recurring runs; genuinely new members are assigned when they first appear.
4. After checkpoints are measured, open the outcome’s record from CAB Outcomes. Under Measurements, read Impact of activation.

The chart places Activated records beside Held back for comparison at the same checkpoint. Future checkpoints remain pending. Select an audience in the ranking to inspect its contribution; if that audience has no control group, CAB says so instead of treating an organization average as its control.

Interpret a positive or negative result

Compare activated and held-back records at the same checkpoint. The following examples show how to read a higher or lower activated rate. Screenshots use synthetic data and show the English interface.

Example, at day 7	Activated	Held back	Difference
Monthly giving started	73 / 404 = 18.1%	8 / 96 = 8.3%	+9.7 percentage points
Lapsed donor reactivated	30 / 303 = 9.9%	15 / 97 = 15.5%	−5.6 percentage points

The first example has a higher observed conversion rate among activated records. In the second, the held-back group did better. Review the sample sizes, the checkpoint, and CAB’s uncertainty or incomplete-reading notices before deciding to expand, change, or repeat an activation. A positive difference alone is not a statistical-significance claim.

Percentage points and lift ratios

A percentage-point difference subtracts the held-back rate from the activated rate. A lift ratio divides the activated rate by the held-back rate. They describe the same comparison in different units.

For example, 12% versus 5% is +7 percentage points and 2.4× the held-back rate, equivalent to a 140% relative increase. It is not a 240% increase. When a lift ratio is shown, a value of 1.0× means equal rates. The ratio needs adequate samples in both groups and a held-back rate above zero; read an unavailable ratio as unavailable, not as zero lift.

The detailed run comparison can distinguish significant, not significant, insufficient sample, and partial, still measuring. These qualify the evidence, not the size of the effect. Later checkpoints may include a different set of mature activations: always compare both arms at the same reading.

Inspect or conclude the experiment

For one activation, expand its addition in Run History. The destination table includes Held out and Measurement where applicable; the expanded view provides the Control group comparison.

Choose View / export control group to inspect Control group (holdout members). This is separate from the activated members shown under Tracked activations.

Use Release control group only when the recurring experiment should end. After confirmation, the next run no longer withholds members: eligible held-back members are added to the campaign and lift measurement stops. Release cannot be undone.

Reserve a control group for one manual add

The ordinary Add to Campaign flow also offers Reserve a control group (%) when Outcome Tracking is available. Enter a percentage before confirming. CAB partitions that one activation and reports how many were held out in its run-history outcome section.

A manual holdout has no standing schedule on which to release members later. Its assignment is independent of another manual add. Use a scheduled experiment when the same people should remain held back across recurring additions and later be included through Release control group.

Experiment guardrails

• Campaign-membership outcomes cannot measure a membership that was never created. Use a compatible conversion-event outcome for the comparison.
• Only one scheduled action may target a campaign while its standing control group is active.
• The percentage locks once the experiment starts.
• The control group excludes members from this campaign only. Other campaigns and interactions can still affect both groups.
• Do not interpret pending readings, failed measurements, or inadequate samples as zero conversion.

For admins

There is no separate control-group switch. Complete Outcome measurement setup, including Outcome Tracking availability, membership-change tracking, the nightly measurement job, and a compatible conversion definition.

Under CAB Settings → Outcome storage, choose how long completed member populations remain available. Retention affects later member inspection and audience filters; it does not turn an organization-average comparison into an experiment.