/bayesian-experiment-reader

name: bayesian-experiment-reader
description: Bayesian counterpart to experiment-result-reader. Computes posterior P(variant beats control), credible intervals, and expected loss from per-variant exposure and conversion data. Beta-Binomial for proportion metrics (CVR), Normal-Normal for continuous metrics (revenue per user). Decision rule combines a confidence threshold with an expected-loss tolerance, so the ship decision reflects both "how likely is this better?" and "how bad is it if I'm wrong?". Use this skill alongside experiment-result-reader when reading any A/B test result. Pairs with analytics-diagnostic-method. Use whenever interpreting an A/B test result the user plans to ship from, when the question is "what's the chance variant wins?", or when a frequentist p-value is on the edge and the user wants the posterior view. Triggers when Clamp MCP returns experiment exposure and conversion data, or when any analytics source surfaces per-variant counts.
when_to_use: When the user asks "what's the probability variant beats control?", "is it safe to ship?", "should we cut our losses?", or when a frequentist read came back ambiguous (p around 0.05) and the user needs a risk-weighted view. Also when stakeholders want posterior probabilities instead of p-values, when the test has been peeked at and frequentist math is contaminated, or when the expected downside of shipping a bad variant is large (revenue tests, churn tests).

Bayesian experiment reader

A frequentist p-value answers a question stakeholders don't ask: "if the variants were identical, how surprising would this data be?" What they actually want is "what's the chance the variant is better?" and "if I ship it and I'm wrong, how bad is it?" Bayesian inference answers both directly. This skill encodes that math and the decision rule it enables.

It pairs with `experiment-result-reader`. Run that one first for the frequentist read and the setup checks (SRM, mix shift, peeking). Run this one to translate the same per-variant counts into a posterior probability and a ship/hold/kill decision.

When NOT to use this

• The setup isn't clean. SRM, exposure-event gaps, or mix shift contaminate Bayesian math just as badly as frequentist math. Fix the setup first via `experiment-result-reader`'s Phase 1 and Phase 4.
• The conversion metric is heavily right-skewed and you only have a handful of conversions per variant (e.g. revenue per user with three whales). The Normal-Normal model assumes approximately normal sampling distributions; small-sample skew breaks it. Either log-transform, bucket into a proportion, or wait for more data.
• The user wants to *design* a new experiment. Sample-size planning under a Bayesian framework is a different problem (expected loss under prior + planned n). This skill reads results, it doesn't plan them.
• The user wants a single number to defend a decision in a hostile review. Bayesian outputs are inherently prior-conditional. If the room won't accept "we used a Beta(1,1) prior," stick with the frequentist read.

Why Bayesian beats frequentist for shipping decisions

Three concrete reasons, not aesthetics:

1. **Posterior probability is the actual decision variable.** "P(variant > control) = 0.97" maps directly to a ship decision. A p-value of 0.03 doesn't: it's the probability of the data under a null hypothesis, which is not what anyone is choosing between. 2. **No peeking penalty.** Bayesian posteriors update coherently as data arrives. There's no alpha-spending budget to blow, no sequential-testing correction required for the math to be valid. (You still want a pre-committed decision rule, but the math itself doesn't degrade.) 3. **Expected loss is the risk side of the ledger.** A variant can be 96% likely to be better and still be a bad ship if the 4% downside is catastrophic (a revenue test where the worst-case is −15%). Frequentist methods don't carry that asymmetry; Bayesian expected loss does.

Method

Phase 1. Confirm the metric type

Two metric types cover almost every A/B test:

| Metric type | Examples | Model | |---|---|---| | Proportion (Bernoulli per user) | CVR, click-through, activation, signup | Beta-Binomial | | Continuous per user | Revenue per user, sessions per user, minutes per user | Normal-Normal |

If the metric is a proportion, use the Beta-Binomial workflow in Phase 2A. If it's continuous, use the Normal-Normal workflow in Phase 2B. If the user is comparing both (e.g. CVR *and* revenue per user), run both independently and report both posteriors.

Phase 2A. Beta-Binomial workflow for proportion metrics

The Beta distribution is the conjugate prior for the Bernoulli likelihood. The update is closed-form: count conversions and failures, add them to the prior's parameters.

**Prior.** Default to `Beta(1, 1)` — uniform on [0, 1], meaning "I have no idea what the conversion rate is." If the user has strong prior knowledge of the baseline (e.g. "our pricing page has converted at 3-4% for 18 months") you can use a weakly informative prior like `Beta(2, 2)` (peaks at 50%, broad) or a baseline-calibrated `Beta(α, β)` where α/(α+β) is the historical rate and α+β is the "pseudo-sample" weight (typically 10–50, not larger — you want the data to dominate).

**Posterior.** For each variant:

posterior = Beta(α0 + conversions, β0 + failures)
         = Beta(α0 + converted, β0 + (exposed − converted))

That's the entire update. No iteration, no MCMC needed for the per-variant posterior.

**Monte Carlo for P(variant > control).** Sample N = 20,000 draws from `Beta(α_v, β_v)` and N draws from `Beta(α_c, β_c)`. Count the fraction where `variant_draw > control_draw`. That fraction is `P(variant > control)`.

**Credible interval on the lift.** Take the 20,000 paired draws of `(variant − control)`, sort them, and read off the 2.5th and 97.5th percentiles. That's the 95% credible interval on the absolute lift in percentage points. (For relative lift, sort `(varian