/19-ab-test-setup-global

name: 19-ab-test-setup-global
description: "Use when the user wants a VALID experiment instead of a guess — hypothesis, one variable, sample size and runtime math, statistical significance, primary versus secondary metrics, multi-arm designs, and a results template, across Optimizely, VWO, and native Meta and Google tests. Trigger on 'A/B test', 'split test', 'how long should I run the test', 'is this result significant', 'test two versions', 'which creative is actually better'. Also use when a winner was declared after two days on tiny numbers. Not for — scaling the proven winner, see `55-scaling-ads-global`; analyzing data already collected, see `13-data-analysis-global`; auditing the account, see `21-ads-audit-global`."
metadata:
  version: 1.0.1
  category: performance
  language: en
license: MIT
triggers:
  - "A/B test"
  - "split test"
  - "multivariate test"
  - "experiment design"
  - "statistical significance"
  - "sample size calculator"
output: A .md file containing hypothesis, sample size calculation, primary/secondary metrics, test setup, timeline, and a results template ready for analysis
related:
  - product-marketing-context-global
  - 13-data-analysis-global
  - 03-performance-eval-global
  - 21-ads-audit-global

A/B Test Setup (Global)

> Run experiments that produce decisions, not noise. Most "A/B tests" in marketing are underpowered, peeked-at, and badly hypothesized — meaning the team learns nothing and ships the louder variant.

---

For Newbies

A valid A/B test answers one question: "Did this change cause a real improvement, or am I seeing noise?"

To answer it credibly you need four things: 1. A **specific hypothesis** with a numeric prediction 2. **One variable changed** (everything else identical) 3. **Enough sample** to detect the effect you care about 4. **Statistical significance** before you call a winner (typically p < 0.05)

If any one of these is missing, you don't have an A/B test — you have a coin flip with extra steps.

**Common newbie mistake:** running a test for 3 days, seeing variant B 40% higher, declaring victory, and shipping. Three days is too short to absorb day-of-week effects, and small samples produce wild swings. Variant B may revert (or reverse) by day 14.

---

Step 0 — Read Context

Read `.agents/product-marketing-context.md` if it exists. Audience size, average traffic, and current conversion rate determine whether a test is even feasible.

---

Step 1 — Information Gathering

Ask up to 4 questions:

1. **What are you testing?** (Ad headline / Landing page section / Email subject / Pricing display / CTA button / Creative video) 2. **Primary metric?** (CTR / Conversion rate / CPM / CPA / Revenue / Open rate / Reply rate) 3. **Daily traffic to the test surface?** (Needed for sample size and duration) 4. **Goal of the test?** (Lift X% on primary metric / Pick a winner among N candidates / Validate a strategic hypothesis)

---

The 7 Principles of a Valid A/B Test

1. Test exactly one variable

The cardinal rule. Change two things at once and you cannot attribute the result.

• **Bad:** "I changed the headline, the hero image, and the CTA color." → You learn nothing about which element drove the lift.
• **Good:** Change only the headline. Image, CTA, layout, traffic source, and audience targeting are identical.

If you must test multiple changes, use a **multivariate test (MVT)** — but those need much more traffic (often 4×–8× a single A/B).

2. Hypothesize with a number

**Format:** "If we [change X], [metric Y] will increase by [Z%] because [reason]."

• **Good:** "If we change the CTA from 'Sign up' to 'Get my free demo,' conversion rate will increase by 15% because action-specific language reduces ambiguity."
• **Bad:** "The new copy will be better." (No metric, no number, no causal reasoning — un-testable.)

The "because" matters: if your hypothesis is wrong but the reasoning was sound, you've still learned something generalizable.

3. Sufficient sample size

Don't stop early. Statistical tests need adequate data to distinguish signal from noise.

• **Minimum rule of thumb:** 100 conversions per variant (not 100 visitors)
• **Better:** Calculate sample size up front based on baseline conversion rate and minimum detectable effect (formula below)

4. Sufficient duration

Run for **whole weeks**, not 3 days, not 10 days. Different weekdays produce different audience behavior — Monday B2B traffic is not Saturday DTC traffic.

• **Minimum:** 7 days
• **Recommended:** 14 days
• **Watch for:** holidays, paydays, monthly billing cycles, ad spend ramp-ups

5. Don't peek

Looking at results every hour and stopping when "B looks good" is the most common error in marketing experimentation. Early peeks combined with early stops dramatically inflate false positive rates.

• Define the end date in advance. Honor it.
• If you must monitor, use **sequential testing** methods designed for it (Bayesian frameworks like Optimizely's Stats Engine, or platforms with built-in sequential controls).

6. Statistical significance: p < 0.05

Most marketing teams use **95% confidence (p-value < 0.05)** as the bar.

• p-value < 0.05 → less than 5% chance the observed difference is random
• p-value 0.05–0.10 → suggestive but inconclusive — extend the test
• p-value > 0.10 → no evidence of an effect — keep control or test something else

For high-stakes tests (pricing, branding) consider 99% confidence (p < 0.01).

7. Document everything

Write down:

• Hypothesis (with number)
• Start date / end date
• Sample size achieved
• Primary metric, secondary metrics
• Result + p-value
• Decision + reasoning
• What you'd test next

A documented test history prevents your team from re-testing things that already failed and from forgetting why you made past decisions.

---

Sample Size Calculation

Quick formula

Sample size per variant ≈ 16 × p × (1 − p) / MDE²

where:
  p   = baseline conversion rate (e.g. 0.03 = 3