/causal-evidence-checklist

name: causal-evidence-checklist
description: Bradford Hill's 9 viewpoints (1965) reframed as a checklist for product analytics. Use this skill before recommending a decision based on observational analytics data. Applies the 9 Bradford Hill viewpoints to score whether X actually caused Y, or whether the correlation is coincidental, confounded, or reversed. Use whenever interpreting a metric change the user is about to act on (rollback, ship, abandon, double-down). Refuses to label a verdict "high confidence" when fewer than ~5 of the 9 criteria pass. Pairs with analytics-diagnostic-method (which provides the hypothesis tree) and channel-and-funnel-quality (which provides the segmentation discipline). Triggers when Clamp MCP returns a comparison the user is about to act on, when a deploy correlates with a metric move, or when the user says "X caused Y" / "did X cause Y" / "should we roll back / ship / kill X" based on a chart. Vendor-neutral methodology; via Clamp MCP the per-criterion checks map directly to traffic.compare, traffic.breakdown, errors.timeline, and funnels.list.
when_to_use: When the user is about to act on a correlation from observational data — "bounce rate jumped after the Tuesday deploy, should we roll back?", "signups doubled after we shipped the new pricing page, can we ship the same treatment to /features?", "churn fell when we launched the new onboarding, did onboarding cause it?". Also when a Clamp MCP comparison returns a directional result and the user is reaching for a decision. Skip when the evidence is from a properly-randomized A/B test (use experiment-result-reader instead — randomization handles most of these criteria for you).

Causal evidence checklist

Observational analytics is full of correlations that look causal and aren't. A deploy ships Tuesday, bounce rate jumps Wednesday, and the instinct is to roll back. Sometimes the deploy did it. Sometimes a marketing campaign landed the same day. Sometimes Wednesday is always like that. This skill encodes a 60-year-old epidemiology rubric — Bradford Hill's 9 viewpoints (1965) — as a checklist the agent fills before recommending an action.

Hill's original audience was epidemiologists deciding whether smoking caused lung cancer without the option of a randomized trial. The same constraint applies to most product analytics: you can't randomize a deploy across a population, so you reason from observational evidence and triangulate. The 9 viewpoints are how.

When NOT to use this

• **The evidence is from a properly-randomized A/B test.** Randomization handles most of these criteria automatically (temporality, specificity, confounding). Use `experiment-result-reader` instead. The checklist is for observational data where you can't randomize.
• **The user only wants an exploratory hypothesis, not a decision.** This skill gates recommendations. If they're brainstorming what *might* explain a chart and are nowhere near acting, it's overkill — use `analytics-diagnostic-method` to build the hypothesis tree first.
• **The metric move is inside noise.** If the "effect" is 1pp on n=200, there's nothing to explain yet. Send the user back to sample-size discipline (in `analytics-diagnostic-method`) before causal reasoning.
• **The system has a known instrumented cause.** If the deploy literally added a `try/catch` around the conversion event and conversions dropped, you don't need 9 criteria — you need to read the diff.

The methodology

Phase 1. State the claim precisely

Write the causal claim as one sentence: "**X caused Y**, where X = [specific change] and Y = [specific metric move]". If the user is vague ("the deploy broke things"), pin them down: which deploy, which metric, over what period, by how much. Without a specific X and Y, the checklist has nothing to score.

Then write the *counterfactual*: "If X had not happened, would Y still have moved?" Most of the 9 criteria are different ways of probing that counterfactual.

Phase 2. Score the 9 criteria

For each criterion, mark **pass / partial / fail** with a one-line justification. The criteria are adapted from Hill's original epidemiological framing to product analytics:

| # | Criterion | Product analytics translation | Pass when... | |---|---|---|---| | 1 | **Strength** | Effect size vs baseline noise | The move is several times larger than the metric's normal day-to-day variance | | 2 | **Consistency** | Same pattern across browsers, geos, devices, time periods | The effect shows up in ≥3 independent slices, not just one | | 3 | **Specificity** | X uniquely causes Y, not a slew of unrelated effects | Y moved but neighboring metrics that *shouldn't* have moved didn't | | 4 | **Temporality** | X preceded Y (non-negotiable) | Y's move starts after X, not before or simultaneous | | 5 | **Dose-response** | Effect scales with exposure/magnitude of X | Heavier-exposed cohorts show larger moves than lighter-exposed ones | | 6 | **Plausibility** | A mechanism makes the link believable | You can name the specific code/UX path that would produce this effect | | 7 | **Coherence** | Fits with what's already known about the system | No prior data contradicts it; adjacent metrics tell a consistent story | | 8 | **Experiment** | Was the hypothesis tested by intervention? | You ran (or can run) an A/B test, holdout, or staged rollback that confirms it | | 9 | **Analogy** | Similar X causes similar Y in adjacent contexts | Past deploys of this shape, or competitor moves, produced the same pattern |

Temporality is non-negotiable: if Y moved before X happened, X did not cause Y. Period. The other 8 are weighted but none is individually decisive.

Per-criterion playbook

**1. Strength.** Compare the move to the metric's own baseline noise. A bounce rate that normally oscillates ±1.5pp daily moving +3pp is suggestive; moving +12pp is loud. Use the prior 14–28 days of the same metric as the noise floor. Via Clamp: `traffic.timeseries` for the metric,