/scanning-experiments-with-replay-vision

name: scanning-experiments-with-replay-vision
description: "Provisions a Replay Vision scanner scoped to one experiment's exposed sessions: sets `experiment_targeting` so the API derives the person-scoped exposure filter server-side, templates a prompt that stays comparable across variants, sizes credit spend against the experiment's own population, and creates the scanner disabled so its prompt can be previewed on real sessions before it sweeps.\nTRIGGER when: user wants Replay Vision to watch an experiment, asks to scan or analyze an experiment's recordings with AI, asks \"what are users actually doing in the test variant\", or wants a scanner scoped to an experiment's exposed sessions.\nDO NOT TRIGGER when: creating a general-purpose scanner not tied to an experiment (use creating-replay-vision-scanners), reading observations a scanner already produced (use exploring-replay-vision-observations), or manually browsing an experiment's recordings without AI analysis (use analyzing-experiment-session-replays)."

Scanning experiments with Replay Vision

The job: _"I'm running an experiment. Watch the recordings and tell me what's actually happening in each variant."_

A Replay Vision scanner is a standing LLM probe over session recordings (see [[creating-replay-vision-scanners]] for the general mechanics). Scoping one to an experiment fixes the classic ways scanners go wrong, all at once: the exposure filter is **derived server-side** from the `experiment_targeting` field instead of hand-authored, the prompt is **templated** from the hypothesis and variants instead of vague, the population is **bounded** by enrollment, and the experiment's end date gives the scanner a natural end. This skill covers what is experiment-specific; the generic create/size mechanics stay in the parent skill.

The flow: resolve the experiment → set `experiment_targeting` so the API derives the exposure filter → pick a template → size it → create **disabled** → preview the prompt on a few real sessions → let the user enable it.

Step 1: Resolve the experiment

`experiment-get` returns everything needed: `feature_flag_key`, the linked `feature_flag` (its `filters.multivariate.variants` list is the source of truth for variant keys — `parameters.feature_flag_variants` can be stale), `exposure_criteria`, `resolved_exposure_event`, `start_date`, `end_date`, and `status`. If the user didn't identify the experiment, resolve it via [[finding-experiments]] rather than guessing.

You no longer derive the exposure event to build the scan query — the API does that from `experiment_targeting` (Step 2). You still need the event name for the per-variant readout join at the end; the readout section covers that derivation where it is used.

Guards before doing anything else:

• **Draft** (no `start_date`): there are no exposures and nothing to scan. Say so and stop.
• **Stopped/complete**: a new scanner only sees sessions from creation time onward, and historical backfill is not automatable over MCP (see Limits). A concluded experiment has nothing left to watch — offer the UI backfill path or a handful of `vision-scanners-scan-session` calls instead.
• **Running or exposure-frozen**: proceed. A frozen experiment stops enrolling but already-exposed users keep producing sessions, so scanning stays useful.
• **Already half over**: the scanner watches only the remaining run. Say so, so a per-variant readout isn't mistaken for full-run coverage.

Step 2: Point the scanner at the experiment

The scanner carries no hand-built exposure filter. Set the `experiment_targeting` field and the API derives the exposure filter server-side, at scan time:

{
  "experiment_id": 123,
  "variant": null
}

• `experiment_id` — the experiment from Step 1.
• `variant` — one variant key to narrow the scan to that variant's exposed people, or `null` for every variant.

**Default to `null` — one scanner for the whole experiment, not one per variant.** The readout splits by variant later (see the readout step). Spend is the same either way — credits are per observation — but one scanner keeps a single prompt version across variants (see Limits on version skew) and one readout, and its random `sampling_rate` is applied _after_ the exposure filter matches, so the sample itself does not bias a variant (the eligibility gates are another story — see Step 4). Set a single `variant` only when the user asks to watch one arm.

**The API owns the exposure filter and its access control.** From `experiment_targeting` the server resolves the same exposed-person population the experiment's Recordings tab shows. The filter is **person-scoped**, so it covers people whose exposure event fired server-side or in an earlier session — sessions the old hand-built event filter missed. You do not build event or property filters for exposure, and you do not run a session-linkability or dedup check first; the server handles both. The API also rejects an `experiment_exposure` set directly inside `query`, and access-checks the targeted experiment, so a scanner can only reach an experiment its editor can view.

**Keep `query` for non-exposure filters only.** Set `filter_test_accounts` from the experiment's own `exposure_criteria.filterTestAccounts`, defaulting to **`false` when absent** — that is what every experiment surface does (`get_test_accounts_filter` backend-side, `?? false` in the replay tab). A minimal query is enough:

{ "kind": "RecordingsQuery", "filter_test_accounts": false }

Add other recording filters (duration, console errors, a specific page) only when the user asks. Leave exposure to `experiment_targeting`.

**No `date_from`/`date_to`.** The scanner strips them on save (its 5-minute sweep controls time) and the estimate ignores them.

Step 3: Scanner type and prompt template

**Default to `classifier`.** A fixed tag set is what makes two variants comparable — free text does not aggregate into a per-variant delta. `scorer` is s