/scanning-experiments-with-replay-vision

name: scanning-experiments-with-replay-vision
description: "Provisions a Replay Vision scanner scoped to one experiment's exposed sessions: derives the recordings filter from the experiment's exposure criteria (with session-linkability checks and honest fallbacks), templates a prompt that stays comparable across variants, sizes credit spend against the experiment's own population, and creates the scanner disabled so its prompt can be previewed on real sessions before it sweeps.\nTRIGGER when: user wants Replay Vision to watch an experiment, asks to scan or analyze an experiment's recordings with AI, asks \"what are users actually doing in the test variant\", or wants a scanner scoped to an experiment's exposed sessions.\nDO NOT TRIGGER when: creating a general-purpose scanner not tied to an experiment (use creating-replay-vision-scanners), reading observations a scanner already produced (use exploring-replay-vision-observations), or manually browsing an experiment's recordings without AI analysis (use analyzing-experiment-session-replays)."

Scanning experiments with Replay Vision

The job: _"I'm running an experiment. Watch the recordings and tell me what's actually happening in each variant."_

A Replay Vision scanner is a standing LLM probe over session recordings (see [[creating-replay-vision-scanners]] for the general mechanics). Scoping one to an experiment fixes the classic ways scanners go wrong, all at once: the query is **derived** from the experiment's exposure criteria instead of hand-authored, the prompt is **templated** from the hypothesis and variants instead of vague, the population is **bounded** by enrollment, and the experiment's end date gives the scanner a natural end. This skill covers what is experiment-specific; the generic create/size mechanics stay in the parent skill.

The flow: resolve the experiment → derive the recordings query from its exposure criteria → pick a template → size it → create **disabled** → preview the prompt on a few real sessions → let the user enable it.

Step 1: Resolve the experiment

`experiment-get` returns everything needed: `feature_flag_key`, the linked `feature_flag` (its `filters.multivariate.variants` list is the source of truth for variant keys — `parameters.feature_flag_variants` can be stale), `exposure_criteria`, `resolved_exposure_event`, `start_date`, `end_date`, and `status`. If the user didn't identify the experiment, resolve it via [[finding-experiments]] rather than guessing.

**Derive the exposure event once and substitute it wherever `<exposure_event>` appears below.** Mirror `get_exposure_event_and_property`: an `exposure_config` explicitly naming `$experiment_exposure` keeps that event on either side of the rollout — the resolved field never overrides an explicit config. Otherwise — no config, or a config naming `$feature_flag_called`, the stored default rather than a custom choice — `<exposure_event>` is the experiment's **`resolved_exposure_event`**, verbatim: the event the analysis counts default exposures on, resolved server-side. Don't re-derive the logic behind it:

• For most experiments today it is plain `$feature_flag_called`. It resolves to the dedicated `$experiment_exposure` event only when **both** hold: the experiment started at or after the rollout cutoff, **and** the team is flagged into the rollout.
• Whichever event resolves covers the experiment's **whole** window — an experiment that predates the new event never resolves to it, so you never query historical sessions on an event that didn't exist yet.
• Ingestion emits `$experiment_exposure` by duplicating flag events, so during the transition both events exist side by side. The analysis counts exactly one — the derived `<exposure_event>` — and so must every query derived here: **never union `$feature_flag_called` with `$experiment_exposure`**.
• For a draft the field reports what the experiment would resolve to if launched now. The field is blind to the exposure config: for an explicit `$experiment_exposure` config or a custom exposure (Case C) it is still populated but must not decide the event — the config wins.

Guards before doing anything else:

• **Draft** (no `start_date`): there are no exposures and nothing to scan. Say so and stop.
• **Stopped/complete**: a new scanner only sees sessions from creation time onward, and historical backfill is not automatable over MCP (see Limits). A concluded experiment has nothing left to watch — offer the UI backfill path or a handful of `vision-scanners-scan-session` calls instead.
• **Running or exposure-frozen**: proceed. A frozen experiment stops enrolling but already-exposed users keep producing sessions, so scanning stays useful.
• **Already half over**: the scanner watches only the remaining run. Say so, so a per-variant readout isn't mistaken for full-run coverage.

Step 2: Derive the recordings query

This is the part that must be right. The exposure criteria define the population; mirror the analysis' semantics rather than inventing a filter shape. The contract is `get_exposure_event_and_property` (`products/experiments/backend/hogql_queries/exposure_query_logic.py`) and its documented frontend mirror `getExposureEventAndProperty` (`frontend/src/scenes/experiments/exposureContract.ts`); `getViewRecordingFiltersForVariant` in `frontend/src/scenes/experiments/utils.ts` shows the filter shape.

**One scanner for the whole experiment, not one per variant.** Pass every variant key and split by variant at readout. Spend is the same either way — credits are per observation, and per-variant scanners would just partition the same sessions (double-scanning any session that fired both variants) — but one scanner keeps a single prompt version across variants (see Limits on version skew) and one readout, and its random `sampling_rate` is applied _after_ the query matches, so the sample itself doesn't bias a variant (the eligibility gates are another story — see Step 4). Only build per-variant