/testing-strategy

name: testing-strategy
description: Use when planning or reviewing tests for a code change, choosing between unit, focused regression, integration, contract, end-to-end, performance, security, property-based, or agent-evaluation checks; classify change risk first and select the smallest evidence set that proves the behavior. Do not use for a single obvious test command, pure documentation changes, or a full security audit without a testing question.

Testing Strategy

Testing is an evidence-selection problem, not a contest to run the largest suite. Choose the smallest set that can falsify the changed behavior, then add one higher-level check only when it covers a boundary the lower level cannot. Keep execution environments reusable, but separate their evidence profiles: `staging-smoke`, `security-proof`, `release-attestation`, and `nightly-stress`. Use `harness-feedback` when a gate is reported as overloaded or misplaced.

Workflow

1. Freeze the acceptance criteria as observable outcomes. 2. Inspect the changed files and classify the risk. 3. Select the lowest useful test level from the matrix below and name the profile. 4. Run the fast gate first. If it fails, fix the cause before adding more tests. 5. Add a focused regression test for a confirmed bug or a changed invariant. 6. Test real boundaries only when the change crosses them. 7. Keep security-proof and release-attestation checks out of staging-smoke unless the acceptance criteria explicitly require that evidence. 8. For high-risk or long-horizon work, use a fresh-context verifier and store the command, revision, result, and skipped checks in a durable artifact. 9. When a verified stage becomes the input to another stage, seal that boundary with commit/tree, contract, input/output digests, and a fresh verdict. Mark an unavailable external prerequisite as `BLOCKED`; do not rerun unrelated accepted code merely because the following environment is unavailable.

Compact Matrix

| Change | Required evidence | Usually deferred | |---|---|---| | Docs, comments, formatting only | Link/lint check when relevant | Runtime suite | | Pure function, local refactor | Fast checks + focused unit/regression tests | Full E2E, mutation | | Parser, serializer, file, DB, API adapter | Fast + focused + one real boundary/integration check | Browser E2E unless user flow changes | | Auth, permissions, migrations, concurrency, public API, deployment | Fast + focused + integration/contract + targeted smoke; independent review for non-trivial changes | Full load test unless performance is in scope | | UI or user journey | Fast + component/focused checks + one stable E2E smoke | Large browser matrix | | Release or performance claim | All applicable lower levels + fixed benchmark/security/release evidence | Nothing that is part of the claim |

The Stop hook runs the project's fast/default suite only when the working tree contains code or test changes. Projects with a complex suite may declare `.claude/test-policy.json`:

{
  "fast": ["python", "-m", "pytest", "-q", "tests/unit"],
  "integration": ["python", "-m", "pytest", "-q", "tests/integration"],
  "release": ["python", "-m", "pytest", "-q"]
}

`fast` is the automatic Stop gate. `integration` is additionally selected for high-risk changes when present. `release` is explicit or CI-only; do not make every edit pay the release-suite cost. Optional `profiles` make the separation explicit; a staging profile must not contain release-signing requirements.

Test Kinds

• **Unit:** isolated behavior and invariants; fast and numerous.
• **Focused regression:** a minimal test that was red before a fix and green

after it. Keep it when it protects a real contract.

• **Integration:** one real boundary such as a database, filesystem, queue, or

external adapter. Use a local/test dependency, never production.

• **Contract:** provider/consumer schema and serialization expectations.
• **Smoke/E2E:** a small number of real user or release paths; keep them stable.
• **Property/fuzz:** invariants over generated inputs; use for parsers,

normalizers, state machines, and edge-heavy algorithms.

• **Performance/security:** only when the change or release claim needs it;

preserve a fixed workload and baseline.

• **Agent eval:** test task completion, tool selection, recovery, and safety on

a versioned golden set. Deterministic assertions come first; an LLM judge is an additional signal, never the sole proof of code correctness.

Agent Evidence Contract

An agent must report: revision, changed scope, commands actually run, exit status, relevant counts, environment constraints, and checks not run with a reason. “Tests passed” without command output or a durable evidence file is not proof. A generated test is a candidate until it reproduces the failure or asserts a stable contract; do not add broad snapshot tests merely to inflate coverage.

For a confirmed bug use `bug-reproducer`: reproduce first, then fix, then run the same test again. For a large/high-risk change use `proof-verify`: a fresh context must produce the final verdict. For a safe structural refactor use `refactoring-safely` and characterization tests before the transformation.

Anti-Duplication Rules

• If a higher-level test finds a failure with no lower-level failure, add the

smallest lower-level reproducer and keep the higher-level test only if it proves a distinct boundary.

• Do not run unit, integration, E2E, benchmark, and security suites by default

just because they exist. Route by changed boundary and risk.

• Do not use retries, sleeps, snapshots, or `skip/xfail` to make red tests look

green. A flaky test needs a cause, a bounded quarantine reason, or a fix.

• Do not claim release readiness from a fast suite alone.
• Do not treat a new external blocker as a failed upstream candidate. Verify the

stage that changed; promote the same sealed input when the prerequisite arrives.

Gotchas

• A g