/scanning-experiments-with-replay-vision

name: scanning-experiments-with-replay-vision
description: "Provisions a Replay Vision `experiment` scanner, which watches one experiment's exposed sessions, records each session's variant, samples the variants evenly, and stops when the experiment ends. Covers status guards, a prompt that stays comparable across variants, sizing credit spend against the experiment's own population, creating it disabled so its prompt can be previewed on real sessions, and reading the results per variant.\nTRIGGER when: user wants Replay Vision to watch an experiment, asks to scan or analyze an experiment's recordings with AI, asks \"what are users actually doing in the test variant\", or wants a scanner scoped to an experiment's exposed sessions.\nDO NOT TRIGGER when: creating a general-purpose scanner not tied to an experiment (use creating-replay-vision-scanners), reading observations a scanner already produced (use exploring-replay-vision-observations), or manually browsing an experiment's recordings without AI analysis (use analyzing-experiment-session-replays)."

Scanning experiments with Replay Vision

The job: _"I'm running an experiment. Watch the recordings and tell me what's actually happening in each variant."_

A Replay Vision scanner is a standing LLM probe over session recordings (see [[creating-replay-vision-scanners]] for the general mechanics). The `experiment` scanner type is built for this job, so use it rather than scoping another type to an experiment:

• The population is the experiment's exposed people, **derived server-side** from the experiment instead of a hand-built filter.
• Each session's variant comes from the exposure data, never from the model. The model is told what the experiment changes, but not the session's variant, so it describes what the recording shows.
• Variants are **sampled evenly**, so a 90/10 rollout still gives the small variant enough observations to compare.
• Scanning **stops when the experiment ends**, and pauses while it is paused.
• Results split by variant with no join: `vision-scanners-variants-list` and the observations `variant` filter.

This skill covers what is experiment-specific; the generic create and size mechanics stay in the parent skill.

The flow: resolve the experiment → write the config → size it → create **disabled** → preview the prompt on a few real sessions → let the user enable it → read the results per variant.

Step 1: Resolve the experiment

`experiment-get` returns everything needed: the linked `feature_flag` (its `filters.multivariate.variants` list is the source of truth for variant keys; `parameters.feature_flag_variants` can be stale), `exposure_criteria`, `start_date`, `end_date`, and `status`. If the user didn't identify the experiment, resolve it via [[finding-experiments]] rather than guessing.

Guards before doing anything else:

• **Draft** (no `start_date`): there are no exposures and nothing to scan. The API refuses an experiment scanner for an experiment that hasn't launched. Say so and stop.
• **Ended**: a new scanner sweeps only from creation onward, and it stops at the end date, so it scans nothing new. Cover the run with a backfill (see Limits).
• **Running or exposure-frozen**: proceed. A frozen experiment stops enrolling, but already-exposed users keep producing sessions, so scanning stays useful.
• **Already half over**: the scanner watches only the remaining run unless you backfill. Say so, so a per-variant readout isn't mistaken for full-run coverage.

Step 2: Write the config

The experiment lives in `scanner_config`, next to the prompt:

{
  "prompt": "Focus on how the user moves through checkout after the payment step first appears: where they hesitate, retry, or leave.",
  "experiment_id": 123,
  "variants": null,
  "balance_variants": true
}

• `experiment_id`: the experiment from Step 1. It is fixed once the scanner exists; watching a different experiment means a new scanner.
• `variants`: the variant keys to watch, or `null` for every variant. **Default to `null`, one scanner for the whole experiment.** A comparison needs every variant, and one scanner keeps one prompt version across them (see Limits on version skew). Narrow it only when the user asks to watch some variants.
• `balance_variants` (default `true`): one sampling rate per variant, so each gets roughly equal coverage for the same spend. Leave it on for comparisons.
• `length` (optional): `short`, `medium` (the default) or `long`, as for a summarizer.

**The API owns the exposure filter and its access control.** It resolves the same exposed-person population the experiment's Recordings tab shows. The filter is **person-scoped**, so it covers people whose exposure fired server-side or in an earlier session. Don't build event or property filters for exposure; the API rejects an `experiment_exposure` set directly inside `query`, and it checks access to the experiment, so a scanner can only reach an experiment its editor can view.

**Keep `query` for non-exposure filters only.** Set `filter_test_accounts` to `exposure_criteria.filterTestAccounts ?? true`, the same rule every experiment surface uses. When the experiment omits the setting, use **`true`**. Keep an explicit `false`, because that is an intentional opt-out. A minimal query is enough:

{ "kind": "RecordingsQuery", "filter_test_accounts": true }

Add other recording filters (duration, console errors, a specific page) only when the user asks. **No `date_from`/`date_to`**: the scanner strips them on save, because its 5-minute sweep controls time.

The prompt

An experiment scanner writes a summary of each session, so the prompt says what the summary should focus on. The scanner adds the experiment's name, hypothesis and the names of the flag's variants by itself, but never the session's own variant: a model told the variant describes what that variant should show, not what the recording shows, and a broken flag gate then goes unseen. Don't r