/ab-testing

name: ab-testing
description: Use when designing experiments for subject lines, offers, cadences, or
  journeys.

Experimentation & A/B Testing Skill

When to Use

• Validating new subject lines or creative.
• Testing segmentation hypotheses (persona vs behavior).
• Optimizing cadence, timing, or automation triggers.

Framework

1. **Hypothesis** – define expected uplift + rationale. 2. **Metric Selection** – primary (open/click/conv) + guardrails (unsubs, spam). 3. **Sample Sizing** – ensure stat significance (min 500 recipients per variant or use power calculator). 4. **Execution** – randomize, keep variants isolated, limit simultaneous tests. 5. **Analysis** – use z-test or Bayesian uplift; document learnings.

Templates

• Experiment brief (hypothesis, segments, KPI, risk guardrails).
• Variant table (control vs test inputs, creative asset links, owner).
• Calculator sheet for minimum detectable effect + sample size.
• Post-test debrief doc capturing learnings + rollout plan.

Experiment Ideas

• Subject line vs preview text combos.
• CTA placement (hero vs footer).
• Personalization depth (basic vs dynamic modules).
• Wait times between touches.

Tips

• Run no more than two tests per journey simultaneously.
• Recycle learnings into playbooks + automation templates.
• Segment results by persona to catch hidden signals.

---