/sequential-monitoring

name: sequential-monitoring
description: Always-valid sequential inference for honest peeking at running A/B tests. Applies mSPRT (mixture Sequential Probability Ratio Test) or confidence sequences so the false-positive rate stays at the nominal alpha even when the test is checked daily. Use this skill when the user asks if it's safe to call an A/B test early, or to peek-check a running test. Pairs with experiment-result-reader and bayesian-experiment-reader, and with analytics-diagnostic-method for the framing discipline. Triggers when Clamp MCP returns mid-experiment exposure and conversion counts, or when a user references peeking, early stopping, sequential testing, alpha-spending, mSPRT, or confidence sequences. Vendor-neutral methodology; works with any analytics source, with Clamp MCP as the canonical integration. Use whenever interpreting an in-flight experiment where the planned horizon has not been reached but the user wants a stop/continue decision.
when_to_use: When the user asks "can we stop the test now?", "the variant looks like it's winning, is it safe to ship?", "I've been peeking, did I break the test?", "what's the sequential p-value here?", or "how do I check a running test without inflating false positives?". Also when an agent is mid-loop on a running experiment and needs to decide whether to call it.

Sequential monitoring

A fixed-horizon A/B test promises a 5% false-positive rate at one planned read. The moment you check the result daily and stop "when it looks good", the actual false-positive rate climbs to 20-30%. Sequential testing fixes this: it lets you check as often as you like and stop the moment the evidence is strong enough, with the type-I error still controlled at the nominal alpha. This skill encodes when to apply mSPRT versus confidence sequences, how to read the boundaries, and when sequential math will not rescue an underpowered test.

When NOT to use this

• The test has fewer than ~400 exposed users per variant. Sequential methods do not manufacture power. At n<400 the boundaries are nowhere near being crossed and the honest answer is "wait, do not peek".
• The conversion metric has strong seasonality (B2B day-of-week, retail weekday/weekend, SaaS payday cycles) and the test has not run a full cycle. Sequential boundaries can cross on a Tuesday and uncross by Sunday; the math is valid but the decision is fragile.
• The user wants to *design* the test (sample size, MDE, variant logic) rather than read a running one. Different skill.
• The experiment was already declared with a fixed analysis plan and the team agreed to read it only at the end. Switching to sequential mid-flight is a governance decision, not a stats one; flag it and ask.

The peeking problem

A fixed-horizon test computes a p-value under the assumption you look once, at the planned end. Each additional peek is another chance to cross the 5% threshold by noise alone.

| Peeks across the test | Actual false-positive rate (nominal 5%) | |---|---| | 1 (end only) | 5% | | 2 | ~8% | | 5 (weekly) | ~14% | | 10 (twice weekly) | ~19% | | Daily over a month | ~25-30% |

If the user has been "checking how the test is doing" every morning, the nominal 5% threshold is meaningless. They need either a sequential method (always-valid) or alpha-spending (pre-commit to a peek schedule and inflate the threshold).

Method

Phase 1. Decide which sequential framework fits

Two practical options. Pick by the decision the user is actually making.

| Question the user is asking | Use | Output | |---|---|---| | "Should we ship treatment or not?" (binary call) | **mSPRT** | Log-likelihood ratio crosses upper or lower boundary → ship or reject | | "What is the lift, with a CI I can trust at any time?" | **Confidence sequences** | Always-valid CI on lift; ship when the CI excludes 0 (or excludes a minimum effect) | | "I just want to peek without breaking alpha" | Either; mSPRT is simpler to read | See above |

Both methods are always-valid: the false-positive rate stays at alpha no matter how often you look. mSPRT is Statsig's default; confidence sequences are GrowthBook's default. The choice is taste plus what the user wants to read.

Phase 2. Verify the setup before peeking

Same prerequisites as `experiment-result-reader`:

1. The experiment is declared (variants, intent, exposure event firing reliably). 2. Per-variant exposure counts are roughly balanced (no sample ratio mismatch). SRM contaminates sequential reads just as badly as fixed-horizon ones. 3. The conversion event is attributable to the exposure session (or windowed appropriately). 4. The smaller arm has n ≥ 400. Below that, sequential math is honest but uninformative.

If any check fails, stop and surface the issue.

Phase 3. Apply mSPRT (binary ship / no-ship)

mSPRT (Johari et al., 2017) generalises Wald's 1945 SPRT to the case where the effect size is unknown. The trick: instead of testing a single point alternative (e.g. "lift = 2pp exactly"), test a *mixture* over plausible alternatives, weighted by a prior. This gives a single log-likelihood ratio statistic that can be checked at every observation.

For a two-proportion test (conversion rate, control vs treatment):

1. **Define alpha and the mixture variance.** Standard choice: alpha = 0.05, mixture variance tau² ≈ MDE² (e.g. if you cared about detecting a 1pp lift, tau ≈ 0.01). 2. **Compute the running log-likelihood ratio.** At each new observation:

   logLR(n) = log( (1 / sqrt(1 + n·V/tau²)) ·
                   exp( (n·V·(diff)²) / (2·(tau² + n·V)) ) )

where `n` is the per-arm sample size, `V` is the pooled variance of the proportion, and `diff` is the observed control-treatment difference. 3. **Boundaries.** Upper boundary = log(1/alpha) ≈ 3.00 for alpha = 0.05. Lower boundary is symmetric (log(alpha)) for two-sided rejection, or use 0 if you only stop for "treatment wins". 4. **Stopping rule.** If logLR > upper boundary → reject H0, s