/diagnosing-experiment-results

name: diagnosing-experiment-results
description: "Diagnoses bias, anomalies, and strange results on a PostHog experiment. Covers 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, surprises after mid-run edits, and qualitative follow-up via a variant-split survey.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping or A/A significance, finds PostHog numbers disagreeing with their SQL, reports surprises after mid-run edits, or wants qualitative feedback or a survey for an experiment.\nDO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle)."

Diagnosing experiment results

This skill answers: **My PostHog experiment results look wrong, biased, or empty — what's going on?**

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged `[HIGH]`, `[MEDIUM]`, or `[LOW]` based on how strongly it's verified — `[HIGH]` is verified directly in PostHog code, `[MEDIUM]` is partially or team-source verified, `[LOW]` describes SDK/external behavior that wasn't verified here. Treat `[LOW]` items as hypotheses to test, not facts to assert.

Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the `finding-experiments` skill first to resolve it to a concrete ID.

Call `experiment-get` and pull these fields. They are inputs for almost every diagnostic:

• `parameters.feature_flag_variants[].rollout_percentage` — the variant split
• `parameters.rollout_percentage` — the overall rollout (% of users entering the experiment)
• `exposure_criteria.multiple_variant_handling` — defaults to `"exclude"` if absent
• `exposure_criteria.exposure_config.event` — unset means the default exposure event; read which one

from `resolved_exposure_event` (`$feature_flag_called` or `$experiment_exposure` — resolved server-side, same properties either way)

• `exposure_criteria.filterTestAccounts` — defaults to `true`
• `feature_flag.active`, status (`draft` / `running` / `paused` / `exposure_frozen` / `stopped`), `start_date`, `end_date`
• `feature_flag.filters.groups[]` — for each group read `variant`, `properties`, and

`rollout_percentage`. Any non-null `variant` is a forced-variant override on the matched cohort (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a variant-pinned group with broad/empty `properties` at high rollout, or no group left randomized (`variant: null`) / no release path to one arm — that starves the other variant (one arm gets ~0 analyzable exposures). See `references/bias-and-skew.md`.

• `stats_config` — Bayesian (default) or Frequentist

Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in [references/diagnostic-snapshot.md](references/diagnostic-snapshot.md). Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.

Step 2 — Match symptom to diagnostic

| User says... | Diagnostic group | | ------------------------------------------------------------------------------------------ | -------------------------------------------- | | "Smaller variant looks biased" / banner says bias | A — bias & skew | | "Variant ratio doesn't match my split" / SRM warning | A — bias & skew | | "Why isn't it 50/50?" / "users in both groups" | A — bias & skew | | "Users in both control and test" / high `$multiple` % | A — bias & skew | | Multi-variant exposure on a server-rendered app | A — bias & skew | | Banner about feature-flag/experiment state mismatch | A — bias & skew | | "Migrating distinct_id" / "switching from anonymous to user_id" mid-run | A — bias & skew | | Metric count is much smaller than exposures (e.g. 10× or 100× gap) | A — bias & skew (route here before D) | | "Experiment shows 0 / not enough data" / empty | B — empty experiment | | "Variant always undefined / false" | B — empty experiment | | "$feature_flag_called fires but no exposures show up" | B — empty experiment | | "Experiment says running but exposures haven't moved in weeks/months" | B — empty experiment | | "Significance keeps flipping as we run longer" | C — interpretation traps | | "Significance was declared, then it wasn't significant anymore" | C — interpretation traps | | "30/16 split at 46 exposures, is this broken?" | C — interpretation traps | | "A/A test is showing significant results"