/experiment-result-reader

name: experiment-result-reader
description: Read the result of a running A/B test honestly. Pulls per-variant exposure and conversion counts, computes lift, applies sequential-testing and sample-size discipline, and surfaces the result in plain language without over-claiming. Built on the experiments section of the event-schema spec; works with any platform that fires a canonical exposure event ($exposure, $experiment_started, or equivalent).
when_to_use: When the user asks "how is X experiment doing?", "did the variant win?", "should we ship this test?", or "is the lift real yet?". Also when comparing two variants of any kind, even outside a formal A/B test, where small-sample noise is the dominant risk. Pair with bayesian-experiment-reader for posterior P(better) and expected-loss decisions, and with sequential-monitoring for honest mid-test peeking (mSPRT / always-valid CIs).

Experiment result reader

A/B tests die from two failure modes that have nothing to do with the variant: reading the result before the sample is big enough, and reading the result without controlling for the wrong thing. This skill encodes the discipline analysts use to avoid both.

When NOT to run this

• The user wants to *design* a new experiment (sample-size estimation, MDE, variant logic). That's a different skill: design vs read.
• The experiment isn't declared in `event-schema.yaml` and exposure events aren't being fired. There's nothing to read; nudge the user toward instrumenting the experiment first via `event-schema-author` or their platform of choice.
• The conversion metric is more than three steps removed from exposure (e.g. "exposure → 30-day retention → upgrade → renewal"). Sequential / cohort-windowed analysis is genuinely harder; this skill stays at the per-period rate level and is honest about it.

Method

Phase 1. Verify the setup

Before reading anything, confirm two facts:

1. **The experiment is declared.** Read `event-schema.yaml`'s `experiments:` section. The experiment should have a name, variants list, and ideally an `intent`. If it isn't declared, ask the user where the variant assignments live and whether the exposure event is firing reliably. 2. **Exposure events are landing for every variant.** Query the analytics platform (with Clamp: `events.list(name="$exposure", group_by="variant")` or whichever event name is canonical for the platform; Mixpanel uses `$experiment_started`, Amplitude uses `$exposure`). You should see roughly even counts per variant if assignment is unbiased; a heavy skew is a setup bug, not a result.

If either check fails, stop and surface the issue. Don't compute lift on a broken setup.

Phase 2. Pull the four numbers

For each variant, get:

• **Exposed users**: unique visitors who fired the exposure event for that variant
• **Converted users**: unique visitors from the exposed set who fired the conversion event in the same period

Conversion rate = converted / exposed, per variant.

The conversion event should be named explicitly by the user (or read from the `experiments[<name>].intent` field if it's in there). Common shapes: `signup_completed`, `subscription_started`, `purchase_completed`, `feature_used`. If the user is vague, ask once.

Phase 3. Compute lift, but caveat it correctly

Lift = (treatment rate − control rate) / control rate.

Three checks before reporting it:

1. **Sample-size threshold.** Use the rule from `analytics-diagnostic-method`: at the smaller variant's `n`, the 95% CI on a proportion is roughly:

| n | ±CI | |---|---| | 100 | 10 percentage points | | 400 | 5 | | 1,000 | 3 | | 10,000 | 1 |

If the observed lift in percentage points is smaller than the CI for the smaller variant, **the result is noise, not signal**. Say so explicitly. Don't report a "12% lift" when the CI is ±15pp.

2. **Run length.** Conversion rates have weekly cycles (B2B is heavier midweek; consumer skews weekends). A test running less than 7 days is structurally unreadable for any conversion event with day-of-week sensitivity. Flag this if relevant.

3. **Peeking penalty.** If the user has been "checking the result" daily, the false-positive rate is higher than the nominal 5%. Don't compute "statistical significance" without acknowledging the user has been watching. Sequential testing math (mSPRT, alpha-spending) is the correct fix; a working approximation is "treat the threshold as 1% nominal if you've been peeking weekly."

Phase 4. Look for what could explain the result *other than* the variant

Even when the lift looks real, two confounders catch most novice analysts:

1. **Mix shift.** Did the variant cohort happen to skew toward higher-converting traffic sources? Slice the conversion rate by `channel` or `device_type` for each variant. If treatment got more organic-search traffic and organic converts higher overall, the "lift" is acquisition-mix, not the variant. Use Clamp's `traffic.breakdown(dimension="channel")` filtered to each variant's exposed cohort to check.

2. **Sample ratio mismatch.** Variants should be ~50/50 (or whatever the assignment ratio dictates). A 60/40 split when assignment was supposed to be 50/50 means either the assignment is broken or the exposure event isn't firing consistently across both arms. Either way the result is contaminated. Compute the ratio explicitly.

If either confounder is in play, the result is unreadable until it's resolved.

Phase 5. Present with the Pyramid Principle

Report the answer first, then the supporting numbers, then the caveats. Specific to experiments:

VERDICT: <one sentence with winner and confidence, OR "too early / inconclusive" with why>

NUMBERS:
- control:    n=<exposed>, conversions=<converted>, rate=<rate>%
- treatment:  n=<exposed>, conversions=<converted>, rate=<rate>%
- absolute lift: <pp>pp
- relative lift: <%>
- 95% CI on lift: ±<pp>pp at the smaller variant's n

CAVEATS:
- <sample-size caveat if applicable>
- <run-length cavea