A campaign can coincide with higher sales without causing them. Experimentation compares outcomes for customers who receive a marketing intervention with outcomes for a comparable control group. That baseline helps separate incremental effects from influences such as seasonal trends or competitor promotions. section 7.7.3 section 7.7.1
Once teams can create and activate audiences quickly, the challenge shifts from launching more campaigns to finding which ones work. Controlled experiments can identify winning audiences and support decisions about customer lifetime value and marketing fatigue. They complement campaign counts and rates, which show performance but do not establish causation, and multichannel models, which guide broader investment decisions. section 3.2 section 7.7.1
Choose a comparison that matches the decision:
- Test messaging: compare a personalized email subject line with a generic one, then examine opens, clicks, and conversions. section 7.7.3
- Test targeting: compare first-party-data lookalike segments with broader audiences to assess return on investment or cost per acquisition. section 7.7.3
- Test a channel: add SMS to an email sequence for one group while another receives email alone; compare conversions or revenue. section 7.7.3
Testing an offer for lapsed customers
A retailer targeted customers who had not opened an email in 60 days with paid-social ads. Short A/B tests compared 10% off with free shipping. Over four weeks, the combined email-and-paid-media effort re-engaged nearly 25% of the dormant audience; the offer tests helped refine the incentive, while that overall result does not isolate the tests’ effect. section 7.2
Build experiments into audience launches rather than treating them as occasional reports. A measurement setup can assign customers to control and treatment groups—or bring existing assignments into campaigns—and connect results to outcomes such as page views, demo views, purchases, or effects on other product lines. Review results, refine segments and messages, and test again. section 4.4.7 section 3.2.2 section 7.7.6
Table 4.8 Standardized measurement framework considerations
| Solution |
Considerations |
| Composable CDPs |
(+) The leading composable CDPs offer experimentation and measurement capabilities, and some can measure outcomes on any metric found in your marketing data lakehouse. |
| Marketing clouds/engagement platforms |
(+/-) MCs will often support experiments, but only on metrics that they host. As of this book’s publication, they do not yet support the ability to run experiments on any metric in your marketing data lakehouse. |
| Monolithic CDPs |
(+/-) Monolithic CDPs will often support experiments, but only on metrics that they host. As of this book’s publication, they do not yet support the ability to run experiments on any metric in your marketing data lakehouse. |
| In-house builds |
(+/-) Longitudinal multivariate experiment frameworks are generally complicated to build, so it is less common for organizations to build them, but global controls and global stratified sampling are certainly possible. |
Compare first-party data solution types by the experiment metrics they can support. The distinction matters when outcomes of interest sit in a marketing data lakehouse rather than inside an engagement platform. section 4.4.7
