/suede-ab-testing

name: suede-ab-testing
description: "Suede AI experimentation discipline: hypotheses, sample size and power, test duration, guardrail metrics, significance and readouts, sequential designs, and an ICE-ranked experiment backlog with a reusable playbook. Use when comparing variants, deciding whether a result is reliable, or building an experiment program and cadence. NOT FOR: analytics instrumentation (use suede-analytics), post-click conversion diagnosis (use suede-site-alchemy), writing the variant copy (use suede-copy), or channel-level incrementality and geo holdouts (use suede-attribution)."
metadata:
  version: 2.1.0

Suede AI A/B Test Setup

Use this experimentation playbook to design tests that produce statistically valid, actionable results.

The Iron Law

Predeclare three things before a test launches: sample per variant,
minimum duration, and the decision rule: and read the result only once
all three are satisfied. A result read before then is preliminary.
Never a winner.

• **Sample per variant**: the Sample Size table below, or a calculator run on your actual baseline.
• **Minimum duration**: 1 full week (day-of-week variation), 2 business cycles (B2B), through paydays (e-commerce): see the "Minimum Duration Rules" section of [references/sample-size-guide.md](references/sample-size-guide.md).
• **Decision rule**: which metric, at which threshold, decides the call: written down before launch, not after.

Two carve-outs, and only these two:

• A **predeclared sequential or always-valid design** may look early under its own stopping rule (see "Sequential Testing" in the sample-size guide). Declaring it sequential after the peek does not count.
• A **guardrail-triggered stop for harm** is a stop, not a winner call. Kill the variant, report no result.

Initial Assessment

Check for `.agents/product-marketing.md` (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md`) and read it if present, baseline conversion rate, traffic volume, and available tooling decide whether a test is even powerable, and they are usually already written down there.

Then work the intake list under Task-Specific Questions below; ask only what the context file did not already answer.

---

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

**Weak**: "Changing the button color might increase clicks."

**Strong**: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

---

Test Types

| Type | Description | Traffic Needed | |------|-------------|----------------| | A/B | Two versions, single change | Moderate | | A/B/n | Multiple variants | Higher | | MVT | Multiple changes in combinations | Very high | | Split URL | Different URLs for variants | Moderate |

---

Sample Size

Quick Reference

| Baseline | 10% Lift | 20% Lift | 50% Lift | |----------|----------|----------|----------| | 1% | 170,000/variant | 43,000/variant | 7,800/variant | | 3% | 54,000/variant | 14,000/variant | 2,600/variant | | 5% | 32,000/variant | 8,200/variant | 1,500/variant | | 10% | 15,000/variant | 3,900/variant | 690/variant |

Two-sided, 95% significance, 80% power, relative lift, rounded up.

**Calculators:**

• Two-proportion sample size calculator: https://www.evanmiller.org/ab-testing/sample-size.html
• Optimizely sample size calculator: https://www.optimizely.com/sample-size-calculator/

**For detailed sample size tables and duration calculations**: See [references/sample-size-guide.md](references/sample-size-guide.md)

---

Metrics Selection

Primary Metric

• Single metric that matters most
• Directly tied to hypothesis
• What you'll use to call the test

Secondary Metrics

• Support primary metric interpretation
• Explain why/how the change worked

Guardrail Metrics

• Things that shouldn't get worse
• Stop test if significantly negative

Example: Pricing Page Test

• **Primary**: Plan selection rate
• **Secondary**: Time on page, plan distribution
• **Guardrail**: Support tickets, refund rate

---

Designing Variants

What to Vary

| Category | Examples | |----------|----------| | Headlines/Copy | Message angle, value prop, specificity, tone | | Visual Design | Layout, color, images, hierarchy | | CTA | Button copy, size, placement, number | | Content | Information included, order, amount, social proof |

Best Practices

• Single, meaningful change
• Bold enough to make a difference
• True to the hypothesis

---

Traffic Allocation

| Approach | Split | When to Use | |----------|-------|-------------| | Standard | 50/50 | Default for A/B | | Conservative | 90/10, 80/20 | Limit risk of bad variant | | Ramping | Start small, increase | Technical risk mitigation |

**Considerations:**

• Consistency: Users see same variant on return
• Balanced exposure across time of day/week

---

Implementation

Client-Side

• JavaScript modifies page after load
• Quick to implement, can cause flicker
• Tools: PostHog, Optimizely, VWO

Server-Side

• Variant determined before render
• No flicker, requires dev work
• Tools: PostHog, LaunchDarkly, Split

---

Running the Test

Pre-Launch Checklist

Each box names the artifact that closes it. An unchecked box means the test is running unvalidated: any result it produces is reportable only as unverified, and a silently broken variant invalidates the entire run's traffic.

• [ ] **Hypothesis documented**: written in the framework structure above, saved with the test record
• [ ] **Primary metric defined**: the metric name plus the predeclared decision rule
• [ ] **Sample size calculated**: n per variant and the projected end date, from the table or a