/experiment-design

name: experiment-design
description: Design and run lean experiments — hypothesis, cheapest test, success criteria, decision. Use when you need to validate before you build.

Experiment Design & Monitoring

The core lean loop: build-measure-learn. This skill structures the full cycle — from shaky assumption to validated learning — so you test cheaply and decide fast. Claude designs the experiment framework and monitors the results. You define what you're betting on and make the call when the data comes in.

When to Use

• You have a hypothesis but aren't sure it's worth a full build
• Leadership wants evidence before committing engineering resources
• You're choosing between multiple solutions and need data, not opinions
• A feature launched but you need to validate it's actually changing behavior
• You want to run an A/B test and need a rigorous setup

When NOT to Use

• You're exploring a problem space — use Discovery Process first
• You need to align stakeholders on whether to experiment — use Stakeholder Alignment
• You're reviewing existing metrics, not testing a hypothesis — use Metrics Review

The AI-Native Approach

| Step | Time | Claude Does | You Do | |---|---|---|---| | Map assumptions | 20 min | Structure assumptions from your context, rank by risk | Validate which assumptions are truly unproven | | Design the experiment | 30 min | Propose cheapest valid test, define metrics and criteria | Confirm feasibility with engineering, set guardrails | | Build the brief | 20 min | Draft experiment brief with hypothesis, method, success criteria | Review and share with the team | | Monitor results | Ongoing | Track metrics, flag significance and guardrails | Interpret results with product context | | Make the call | 15 min | Summarize findings, frame the decision | Decide: ship, iterate, or kill |

Process

Step 1: Map Assumptions (20 minutes)

Every product bet is a stack of assumptions. Most teams skip straight to building. Lean teams identify which assumptions are riskiest and test those first.

We're considering [initiative/feature]. Here are the assumptions behind it:

List every assumption we're making:
- About the user problem (is it real? how painful?)
- About the solution (will this approach work?)
- About the market (will people pay? switch? adopt?)
- About feasibility (can we build it in time? at this cost?)

For each assumption:
- How confident are we? (high / medium / low)
- What's the consequence if we're wrong? (high / medium / low)
- Is there existing evidence? (data, research, analogues)

Rank by risk: low confidence + high consequence = test this first.

Step 2: Design the Experiment (30 minutes)

The goal is the cheapest, fastest test that gives you a decision-quality signal. Not a perfect study — a lean experiment.

For our riskiest assumption: [assumption]

Design the cheapest valid experiment:

Hypothesis: "We believe [this] will cause [that] for [whom]."
Null hypothesis: "There is no meaningful difference when [this] changes."

Method options (from cheapest to most expensive):
1. Fake door / painted door test
2. Concierge / manual-behind-the-scenes
3. Wizard of Oz prototype
4. A/B test on existing feature
5. Limited beta / feature flag rollout
6. Full A/B test on new feature

For the recommended method:
- What do we build/change? (minimum implementation)
- Who do we expose it to? (segment, sample size)
- How long do we run it? (minimum duration for significance)
- What do we measure? (primary metric, secondary, guardrails)

Success criteria:
- Ship if: [primary metric] improves by [X%] with [confidence level]
- Iterate if: [signal is positive but below threshold]
- Kill if: [no signal or guardrail metric degrades by Y%]

Guardrail metrics (must not degrade):
- [Metric 1]: [threshold]
- [Metric 2]: [threshold]

Step 3: Build the Experiment Brief (20 minutes)

Draft an experiment brief:

1. Background: Why we're running this experiment (1-2 sentences)
2. Hypothesis: Stated as testable prediction
3. Method: What changes, for whom, for how long
4. Primary metric: What we measure, target, baseline
5. Secondary metrics: Supporting signals
6. Guardrail metrics: What must not degrade
7. Sample size and duration: Minimum for statistical significance
8. Success criteria: Ship / iterate / kill thresholds
9. Decision date: When we read the results and decide
10. Owner: Who makes the call

Keep it to one page. If the brief is longer than the experiment, you're over-engineering it.

Step 4: Monitor Results (Ongoing)

Here are the current experiment results: [paste data or describe]

Analyze:
- Are we reaching statistical significance? How many more days at current rate?
- Is the primary metric trending toward the success threshold?
- Are any guardrail metrics showing degradation?
- Are there segment-level differences worth investigating?
- Any external factors that could be confounding results?

Flag:
- 🟢 On track — continue running
- 🟡 Mixed signals — may need more time or investigation
- 🔴 Guardrail breach — consider stopping early
- ✅ Reached significance — ready for decision

Step 5: Make the Call (15 minutes)

The experiment has concluded. Here are the final results: [data]

Summarize:
1. Did the primary metric hit the success threshold?
2. Did any guardrail metrics degrade?
3. Were there surprising segment-level differences?
4. What did we learn that we didn't expect?

Frame the decision:
- Ship: results met criteria, here's the rollout plan
- Iterate: signal is there but we need to adjust [what] and retest
- Kill: no meaningful signal, here's what we learned and what to try next

Either way: what's the validated learning? What do we now know that we didn't before?

Output

• Ranked assumption map with risk scores
• Experiment brief (hypothesis, method, metrics, success criteria)
• Monitoring summary with significance tracking
• Decision recommendation with validated learning

Com