/ln-72-product-outcome-evaluator

name: ln-72-product-outcome-evaluator
description: "Evaluates observed product outcomes against a prior hypothesis; does not run experiments or change user treatment."

Product Outcome Evaluator

**Goal:** Determine what available evidence supports about a delivered product outcome and recommend continuation, adjustment or stopping. Remain read-only: do not change instrumentation, experiments, user treatment, campaigns or product files.

**Execution contract:** The checklist defines completion. Track each item internally as `PENDING`, `PROVEN` with evidence, `CLEARED` with evidence its condition is absent, or `UNPROVEN` with a gap; reading, delegation, or tool failure is not proof. Reconcile after each section. Before returning, resolve all `PENDING`, count only `PROVEN` and `CLEARED`, and apply verdict and approval rules to every gap. Preserve intent, scope, and existing authorization. Continue authorized work; ask only for consequential unresolved choices or required external approval. Scale depth to material risk without skipping checks. Preserve dependency and safety order; otherwise choose an appropriate verification method. Accept equivalent user or repository evidence; no other skill, named artifact, or complete lifecycle is required. Preserve source requirement and decision IDs. Bind reused evidence to relevant source versions, dirty changes, configuration, and environment; invalidate only affected claims. On continuation, reconcile task, authorization, current state, and unresolved evidence. For long work, return a compact continuation record or update an already authorized artifact; read-only skills do not persist it. Distinguish artifact readiness, verified behavior, and external-action authority. Prepare authorized work before required approval. If blocked by an instruction, cite its exact source and unresolved boundary; do not invent approval gates from caution.

Tool Routing

| Need | Preferred capability | Fallback | |---|---|---| | Original hypothesis | Product intent, baseline, experiment/measurement plan and accepted targets | Reconstruct from attributable sources; keep missing targets unknown | | Outcome evidence | Authorized analytics, experiment results, customer behavior and cost/support evidence | Sanitized exports with explicit measurement limits | | Analysis | Reproducible queries/statistics appropriate to the study design | Transparent arithmetic and qualitative inference; no fabricated causal confidence |

Domain Rules

• Distinguish delivered behavior, observed metric movement and causal product impact. A release or acceptance test proves neither adoption nor business value.
• Do not choose success thresholds after seeing the result. Separate predeclared criteria from exploratory findings and owner preferences.
• Use only authorized data with necessary minimization. A recommendation is not permission to run an experiment or contact users.

Checklist

1. Frame the Outcome Decision

• [ ] Resolve the delivered capability, intended audience, original hypothesis, decision horizon and outcome decision requested.
• [ ] Identify the released/deployed version, rollout/exposure window and relevant baseline or comparison group.
• [ ] Recover predeclared primary metrics, guardrails, targets and stop rules; mark absent criteria rather than inventing them.
• [ ] Separate product intent and owner preference from measured behavior and external assumptions.

2. Assess Measurement Fitness

• [ ] Inspect metric definitions, units, denominators, event coverage, deduplication, identity joins and missing data.
• [ ] Check whether users were actually exposed and whether observation duration supports the intended outcome.
• [ ] Assess cohort composition, selection bias, seasonality, concurrent changes and other confounders.
• [ ] For experiments, inspect assignment, contamination, sample imbalance and uncertainty using the actual study design.
• [ ] Distinguish trustworthy measurements, reported results, estimates, qualitative signals and unavailable evidence.

3. Evaluate Value and Harm

• [ ] Compare outcomes with valid baselines or controls using reproducible calculations and appropriate uncertainty.
• [ ] Check guardrails and material regressions in user experience, reliability, support burden, cost or data quality.
• [ ] Separate aggregate effects from relevant segments and expose tradeoffs without fishing for favorable subgroups.
• [ ] Distinguish causal conclusions supported by the design from correlations and exploratory interpretations.
• [ ] Identify whether failure lies in adoption, interaction, correctness, measurement or the original value hypothesis.

4. Recommend the Next Decision

• [ ] Recommend continue, adjust or stop only to the degree supported by the evidence; explain what could reverse the recommendation.
• [ ] For uncertainty, define the cheapest next measurement or experiment with audience, signal, boundary and decision criterion without executing it.
• [ ] Return results linked to the original requirement/hypothesis and observed deployment state.
• [ ] Report data and causal limitations explicitly; do not transform lack of proof into proof of no effect.

Verdict

• `SUPPORTED`: evidence supports the intended outcome within the stated population, window and causal limits.
• `NOT_SUPPORTED`: valid evidence contradicts the declared outcome or violates a required guardrail.
• `INCONCLUSIVE`: evidence cannot establish the outcome or causal interpretation.
• `BLOCKED`: essential hypothesis, exposure identity or authorized data is unavailable.

Self-Check

• [ ] **Reconcile before returning.** Check item-level evidence, requirement coverage, contradictions, scope, verdict, and applicable cleanup. Correct the report or authorized artifacts. Reuse valid evidence; do not automatically rescan the repository or rerun successful commands. Repeat checks only for relevant changes, failures, or unresolved evidence. Disclose remaini