Experiments and engagement

Work with the right context

Declare what the experiment measures before splitting the audience. Keep variant recipients disjoint and preserve the audience snapshot, observation window and sample requirement. The team should be able to explain why a winner is supported or why the result remains inconclusive.

Keep the next action clear

Engagement signals have different meanings. A proxy open does not prove that a person read a message, and a click may require filtering before it supports a decision. Retain those distinctions when comparing content and selecting a follow-on variant.

Coordinate the workflow

Process delivery events. Authenticate and deduplicate each provider event by its event ID. Several opens or clicks may refer to one message without replacing each other. Preserve out-of-order event history and reconcile terminal delivery facts without moving a failed recipient back to Queued. Label proxy/bot/unknown engagement so open rates do not claim verified human attention.

Review experiment. Assign recipients to disjoint variant groups using a frozen audience snapshot. Use the declared metric, sample requirement and decision window. Campaign Approver independently accepts a supported winner or records Inconclusive; no automatic winner is invented when evidence is insufficient. A follow-on send targets only the untouched eligible remainder. Supported metrics are OpenRate, ClickRate and ClickToOpenRate with disclosed signal quality. There is no conversion-attribution metric; verified CRM events used to exit a journey do not count as attributed experiment conversions.

Review campaign outcomes. Compare accepted, delivered, bounced and suppressed recipients as distinct measures. Deduplicate event counts and distinguish proxy-derived engagement. Review acquisition and preference evidence against the configured jurisdiction and purpose policy; no template or report certifies legal compliance. Propose new content and audience versions from observed outcomes.

A/B testing. The approved winning variant reaches only the untouched remainder. Insufficient evidence stays Inconclusive; reject unsupported Conversions metric, automatic winner dispatch and a recipient receiving multiple variants.

Engagement tracking. Distinct clicks on one message remain distinct events without replay inflation. Duplicate callbacks do not inflate counts and proxy opens are not labeled verified human activity.

Campaign analytics. Reported totals reconcile to unique recipient and provider event records. Do not equate provider acceptance with delivery or claim inbox placement from opens.

Compare variants without overstating the result

An experiment assigns eligible recipients to disjoint variants before the test begins. Record the audience, comparison metric and required evidence so the team knows what the result can support. A recipient should not move between variants during the same experiment, because that would blur which version generated the observed outcome.

Email opens and clicks need interpretation. A proxy or security scanner may fetch content without a person reading it. Preserve the signal quality and deduplicate event occurrences, so a repeated callback or automated fetch cannot silently inflate a human engagement claim.

Decide whether the test supports a winner

Review the evidence against the configured decision rule. When the evidence is insufficient, retain that outcome and do not select a winner merely because one displayed number is higher. The campaign approver can inspect the comparison and the proposed next action together.

A reviewed winner can apply to the untouched remainder of the eligible audience. It should not resend a winning variant to people who already received a test message. Continue checking current preferences and suppression for that remainder at dispatch time. The outcome is a comparison the marketing team can explain: who was included, which version they received, which events were reliable and whether the evidence justified the next send.