/debugging-experiments

name: debugging-experiments
description: >-
  Debug and support PostHog Experiments (A/B tests) for a customer looking at
  their own results. Use whenever an experiment support ticket is pasted or a
  customer asks a results question, most commonly "why aren't my exposures
  even?", "why is one variant getting no traffic?", "why am I missing / seeing
  too few exposures?", "why does the bias banner show?", or "why don't PostHog's
  numbers match my SQL?". Pulls the experiment's real data read-only, matches it
  to a known-cause catalog, and produces a customer-facing explanation, fix, and
  review of the pertinent numbers. Loads diagnosing-experiment-health as its
  deep diagnostic library.
  DO NOT TRIGGER when: creating an experiment (use creating-experiments),
  only configuring rollout (configuring-experiment-rollout) or metrics
  (configuring-experiment-analytics), asking lifecycle questions
  (managing-experiment-lifecycle), or the underlying feature flag is what's
  misbehaving rather than the results (use debugging-feature-flags).

Debugging experiments

PostHog Experiments are A/B tests: a feature flag randomizes users into variants, the SDK records an **exposure** when the flag is read, and PostHog computes per-variant metrics and significance. A customer looks at that results page and asks why it looks wrong.

**Most experiment-results tickets are config or exposure-collection problems, not statistics bugs.** The randomization is fine; something upstream is skewing which users get exposed, or stopping exposures from being recorded. The job is to find _which_, prove it with the customer's own data, and hand back a plain-language explanation plus the fix.

This skill is the customer-support front door. It carries the two most common complaints inline (uneven exposures, missing exposures) and loads [`diagnosing-experiment-health`](../diagnosing-experiment-health/SKILL.md) as a diagnostic library for the deeper long tail (interpretation traps, numbers-vs-SQL, mid-run surprises).

Debugging workflow

1. **Parse the ticket.** Extract project ID, instance (US vs EU — the URLs and data live in different places), experiment ID or name, the `lib`/platform if relevant, the exact complaint in the customer's words, and what they already tried. Aged or multi-reply tickets are dirty: the config may have been edited mid-thread, so re-pull current state and treat earlier claims as stale. 2. **Resolve the experiment.** If the ticket names it rather than giving an ID, load [`finding-experiments`](../finding-experiments/SKILL.md) to resolve it, then call `posthog:experiment-get`. 3. **Pull the data read-only.** Run the fixed data-pull sequence in [references/pulling-the-data.md](references/pulling-the-data.md). This produces the "pertinent numbers" you will show the customer: per-variant exposed-person counts, `$multiple` share, the `distinct_id`/`person` fragmentation ratio, the SRM chi-squared result, the exposure trajectory, and the flag/experiment activity log. Verify from data before asking the customer anything. 4. **Match the complaint** to the known-cause catalog below. Confirm the single leading cause with one targeted number from step 3 before writing. Treat the customer's _own_ conclusion ("it's just noise", "a measurement bug") as a hypothesis to **disconfirm**, not confirm — pull the data independently rather than re-deriving their answer. Quantify a suspected cause before asserting its impact (count the contaminating cohort, don't eyeball it). One trap in particular: never run the SRM chi-square against an _assumed_ even split — read the configured `rollout_percentage` first, since an intended 34/33/33 reads as a ~2% SRM under an equal-split assumption. 5. **Scope the fix to the experiment's state** before recommending it. On a **draft**, config changes are free — recommend freely. On a **running** experiment every change has a mid-run tradeoff (changing the split is an anti-pattern — prefer reset or end+restart; see [`configuring-experiment-rollout`](../configuring-experiment-rollout/SKILL.md) and [`managing-experiment-lifecycle`](../managing-experiment-lifecycle/SKILL.md)). On a **stopped/shipped** experiment the flag and results are the documented outcome, so recommend interpretation or a _next_ experiment, not a mid-run edit. Don't propose reversing a state change unless the customer asks how to undo it. 6. **Write the reply** using [references/customer-reply.md](references/customer-reply.md): cause → fix → the numbers that prove it, in the customer's UI language.

Known-cause catalog — "exposures aren't even" / "one variant has no traffic"

Ordered by how often they're the answer. Full mechanism detail lives in [`diagnosing-experiment-health/references/bias-and-skew.md`](../diagnosing-experiment-health/references/bias-and-skew.md) (group A) — load it when a case needs more depth than the summary here.

**First, split a real SRM into its two possible homes.** Assignment is a deterministic hash of a stable identifier (the `distinct_id` by default; the device ID or group key for those flag types — see [references/pulling-the-data.md](references/pulling-the-data.md)), so with an unchanged split every user has a _fixed_ variant and any set of users must fall close to the configured percentages. A confirmed SRM (chi-squared p < 0.001 at healthy volume — not eyeballed) therefore lives in exactly one of two places:

• **Assignment-side** — the recorded variant disagrees with what the hash would assign. Something

overrode assignment at serve time: a stale local-evaluation definition, an inherited bootstrap value, a forced release-condition variant, or a mid-run rehash.

• **Capture-side** — the recorded variant _agrees_ with the hash (assignment is fine), but _which_

users get an exposure recorded is selected: one arm reaches a surface the other never does, or one arm's users read the f