/test-value-evidence

name: test-value-evidence
description: >-
  Assess tests for an identified code or test change, feature, PR or commit. USE FOR: review
  changed behavior or regression tests, prove pre-fix/missing behavior is
  detected, revision-bound red-green evidence. Polyglot. DO NOT USE without an
  identified change: hypothetical mutations or general suite blind
  spots (test-gap-analysis). Also exclude per-test readiness/A-F grades (grade-tests),
  coverage (coverage-analysis), new suites (code-testing), or mutation engines.
license: MIT

Test-value evidence

A passing test proves little unless it detects the relevant wrong behavior. Assess the changed contracts and their directly related tests, not the entire repository. Discover and follow the repository's instructions, pinned toolchain, test commands, assertion conventions, and execution permissions. This skill does not authorize edits in a read-only review, execution of untrusted code, workflow dispatch, publication, or access to another session's checkout.

Inputs and boundaries

Discover the requested diff/revision, production entry points, tests, fixtures, and available execution artifacts. If the change cannot be identified, report the missing scope instead of grading the workspace. Existing unchanged tests can suffice; a source-only diff is not automatically untested.

Use `test-gap-analysis` for general existing-suite blind spots, `grade-tests` for curated per-test readiness/quality, and existing repository mutation tooling for mutation campaigns. Do not start a second mutation engine or require full-suite mutation testing for every change. This skill is self-contained: sibling skills are optional routing destinations, not required dependencies.

Workflow

1. **Bind the contract to the tested revision.** For a PR, record its merge base and exact head SHA. For a commit, record the selected parent and commit; identify the parent explicitly for a merge commit. For uncommitted work, record the base SHA plus a reproducible patch identity (saved patch/hash, including relevant untracked files), not a fictitious committed revision. Evidence expires after relevant source, tests, helpers, configuration, or package changes unless revalidated.

For each distinct behavior change, map: `requirement -> input/sequence -> expected public outcome -> production entry point -> named test/data case -> distinguishing assertion`. Derive the expected outcome independently from requirements or supported API/protocol semantics, not the implementation or the test's current value. Check build/package/configuration behavior, deleted/renamed code, removed tests, and shipping consumers as well as source. Include positive, negative, boundary, compatibility, cancellation, cleanup, and multi-module cases only where relevant to the changed contract.

A mock of the changed component, self-comparison, or expected value computed by the same production algorithm does not provide an independent oracle.

2. **Choose the meaningful counterfactual.**

| Change | Counterfactual | Required observation | |---|---|---| | Bug fix | Keep the regression test and restore the actual pre-fix defect. | The test fails for that defect, then passes with the fix. | | New feature | Disable/remove or corrupt the behavior, retaining the API shape and setup needed to build. | An assertion detects absent/wrong behavior, then passes with the feature. | | Stronger/test-only coverage | Introduce one plausible wrong public outcome. | The strengthened test detects it; the unmodified product passes. | | Behavior-preserving refactor | Compare the preserved contract before and after; optionally probe a relevant fault. | Preservation tests pass on both revisions; passing before is expected. | | Documentation/generated metadata only | Establish whether an executable contract changed. | Not applicable only with a concrete reason; generated or build metadata can change behavior. |

For a fix, prefer the real pre-fix implementation with the same regression test, setup, dependencies, and inputs. If new APIs prevent that combination from compiling, restore only the defective behavior in the head version. Call this a **targeted counterfactual**, not a run on the base revision. Record its exact patch/hash. Never change assertions to manufacture red.

For a feature, a missing-symbol compiler error is not behavioral proof. Choose a buildable no-op, disabled registration, old branch, or wrong output. A compiler/analyzer diagnostic can be valid behavior only when an executed harness asserts that exact diagnostic or its absence.

Prefer small plausible faults: restore the old boundary, drop a required output, skip a callback, ignore an option, or return the wrong value. An equivalent mutation is not a gap. A crash, timeout, or exception counts only when it is the intended contract signal and diagnostics attribute it to the fault, not setup, discovery, flakes, or infrastructure.

3. **Execute paired runs only when authorized.** Implementation normally requires execution; read-only/advisory review normally requires inspection. A review request alone does not permit source edits or builds/tests. Never run untrusted code with privileged credentials.

In an authorized implementation/verification environment:

• Use an owned isolated checkout or experiment directory. Preserve the

user's dirty files and other sessions/main checkout. Preflight concurrent processes and shared outputs/resources; do not kill unrelated processes.

• Run the smallest relevant selection on the correct implementation.

Confirm the named tests and data cases actually execute and pass.

• Apply only the chosen defective behavior in the isolated experiment.

Rebuild and rerun the same tests, assertions, data, dependencies, and applicable runtime configuration. Capture the named failure