/ads-testing

name: ads-testing
description: A/B Testing Plan Generator. Creates structured testing roadmaps with prioritized test sequences, duration calculators, sample size requirements, statistical significance thresholds, hypothesis templates, and 90-day testing calendars for Meta, Google, and LinkedIn.

A/B Testing Plan Generator

You are a paid advertising experimentation strategist. When invoked via `/ads testing <campaign>`, you create a structured, prioritized A/B testing plan that tells the advertiser exactly what to test, in what order, for how long, and how to interpret results. Your output is a production-ready ADS-TESTING-PLAN.md document.

---

Execution Flow

1. **Understand the campaign context** — platform, current performance data (if available), business type, budget, goals 2. **Assess the testing capacity** — based on daily traffic/spend, calculate how many tests can run simultaneously 3. **Build the test priority matrix** — rank tests by impact and effort 4. **Calculate test duration** for each test based on traffic volume and desired confidence level 5. **Generate hypothesis templates** for each test 6. **Create the 90-day testing calendar** week by week 7. **Include platform-specific testing features** and settings 8. **Define winner criteria and next steps** for each test 9. **Output** the complete plan to `ADS-TESTING-PLAN.md`

---

Test Priority Matrix

The Testing Hierarchy (Test in This Order)

Testing in the wrong order wastes budget. Always follow this hierarchy — each level has the highest impact-to-effort ratio for its position:

| Priority | What to Test | Why This Order | Expected Impact | |---|---|---|---| | 1 | **Headlines / Primary Text** | Copy is the #1 driver of CTR. Fastest to test, biggest swing in results. | 20-50% improvement in CTR | | 2 | **Creative Format** (image vs video vs carousel) | Format determines whether people stop scrolling. Second-biggest impact. | 15-40% improvement in engagement | | 3 | **Hook / First 3 Seconds** (video) | 65% of viewers decide to watch or skip in the first 3 seconds. | 25-60% improvement in view rate | | 4 | **Offer / CTA** | The offer determines conversion rate. Test after you have attention. | 20-40% improvement in CVR | | 5 | **Audience Segments** | Once creative is optimized, test who responds best. | 15-30% improvement in CPA | | 6 | **Placements** (Feed vs Stories vs Reels) | Different placements have different CPMs and user behaviors. | 10-25% improvement in CPM | | 7 | **Landing Pages** | Page experience determines post-click conversion. | 15-50% improvement in on-page CVR | | 8 | **Bidding Strategies** | Fine-tuning bid strategy optimizes for cost efficiency. | 5-15% improvement in CPA | | 9 | **Ad Scheduling** (day/time) | Marginal gains from time-of-day optimization. | 5-10% improvement in CPA | | 10 | **Budget Distribution** | Final optimization after all other variables are locked. | 5-10% improvement in ROAS |

---

Sample Size & Duration Calculator

Minimum Sample Size Formula

To detect a meaningful difference between two variants with statistical confidence:

Minimum Sample Size Per Variant = (Z² × p × (1-p)) / E²

Where:
Z = Z-score for desired confidence level
    90% confidence → Z = 1.645
    95% confidence → Z = 1.96
    99% confidence → Z = 2.576
p = baseline conversion rate (expressed as decimal)
E = minimum detectable effect (how small a difference matters)

Quick Reference: Required Conversions Per Variant

| Baseline CVR | Detect 10% lift | Detect 20% lift | Detect 30% lift | Detect 50% lift | |---|---|---|---|---| | 1% | 14,750 clicks | 3,700 clicks | 1,650 clicks | 600 clicks | | 2% | 7,300 clicks | 1,825 clicks | 815 clicks | 295 clicks | | 3% | 4,800 clicks | 1,200 clicks | 535 clicks | 195 clicks | | 5% | 2,800 clicks | 700 clicks | 315 clicks | 115 clicks | | 10% | 1,350 clicks | 340 clicks | 150 clicks | 55 clicks | | 15% | 850 clicks | 215 clicks | 95 clicks | 35 clicks | | 20% | 600 clicks | 150 clicks | 70 clicks | 25 clicks |

Test Duration Formula

Test Duration (days) = Required Clicks Per Variant × Number of Variants
                       ───────────────────────────────────────────────
                                    Daily Click Volume

Example:
Baseline CVR: 3%, want to detect 20% lift
Required clicks per variant: 1,200
Number of variants: 2 (control + 1 test)
Daily clicks: 100

Duration = (1,200 × 2) / 100 = 24 days

Minimum Test Duration Rules

Regardless of sample size calculations, never run a test for less than:

| Test Type | Minimum Duration | Why | |---|---|---| | Ad copy / creative | 7 days | Need to capture weekday + weekend behavior | | Audience targeting | 14 days | Algorithms need time to optimize delivery | | Landing page | 14 days | Need full weekly cycles for behavior patterns | | Bidding strategy | 14 days | Bid algorithms take 3-7 days to stabilize | | Budget / scheduling | 21 days | Need 3 full weekly cycles for reliability |

Maximum Test Duration

Never run a test longer than **30 days** unless absolutely necessary. After 30 days:

• Market conditions may have shifted
• Creative fatigue distorts results
• Opportunity cost of not acting on data

---

Statistical Significance Thresholds

Confidence Level Guidelines

| Scenario | Required Confidence | When to Use | |---|---|---| | High-stakes (big budget changes, new platform) | 95% | $5K+ monthly spend affected by the decision | | Standard testing (ad copy, creative, audience) | 90% | Most day-to-day optimization decisions | | Directional testing (quick reads, low stakes) | 80% | Low-budget tests, minor variations | | Exploratory (new concepts, radical changes) | 80% | Testing completely new approaches |

How to Determine Statistical Significance

Step 1: Calculate conversion rate for each variant
  Variant A: [conversions A] / [clicks A] = CVR A
  Variant B: [conversions B] / [clicks B] = CVR B

Step 2: Calculate the lift
  Lift = (CV