/ia-test-audit

name: ia-test-audit
class: workflow
description: >-
  Audit whether tests detect regressions in the behavior they claim to protect.
  Find mocked-away subjects, weak or circular assertions, undiscriminating fixtures,
  swallowed failures, and tests missing from gates. Use for test-quality audits,
  suspected false confidence in generated tests, and reviewing new tests. Use
  ia-writing-tests for writing tests.

Test audit

Find tests that remain green when their claimed behavior breaks. Assess behavior, not apparent authorship, mock count, assertion count, or a target finding percentage. Default to a read-only audit. Change tests or production code only when requested.

Adapted from the OpenClaw test-audit skill (MIT).

What counts as a finding

Evaluate each claimed behavior separately. A test can protect one obligation and miss another. Use these dispositions:

• **Ineffective for the claim:** the named behavior is replaced, never reached, or

disconnected from a verdict that can fail.

• **Partially effective:** real behavior is tested, but a specific promised distinction

is absent from the input or assertions. Preserve the protection that exists.

• **Not enforced:** the test is uncollected, skipped, or absent from a blocking gate.

Report local collection and CI enforcement separately from assertion quality.

• **Redundant:** another test protects the same behavior at the same boundary under

equivalent inputs and setup. This requires separate evidence; weakness alone is not redundancy.

• **Adequate for its scope:** the test detects a credible failure of its actual contract.
• **Unresolved:** missing context or execution prevents a defensible decision.

A missing additional edge case is not automatically a defective existing test. Tie each quality finding to an existing claim, documented contract, or misleading assertion. Distinguish an individual test's weakness from a suite-wide coverage gap.

1. Establish scope and execution

Read repository instructions, runner configuration, shared fixtures, test helpers, and CI commands. Record the exact focused command and applicable completion gates. Determine which tests are collected, execute, and block a merge. An advisory job is not a blocking gate. A local green result does not establish CI enforcement.

Build an inventory from the runner's collection output when practical. Otherwise label the source inventory approximate. Count files, test definitions, and parameter cases separately. Do not treat helpers as tests or silently omit unrecognized syntax.

For a bounded request, inspect every in-scope test. For a large suite, declare the review budget and selection method before discovery. Cover distinct layers, major fixtures, and mocking styles. Combine risk-directed selection with a reproducible sample independent of detector hits. Keep those two sample results separate. Record the seed and sample unit if claiming random sampling. Expand recurring patterns into their owner family when practical; report anything left unreviewed.

Do not infer suite prevalence from targeted examples. If a prevalence estimate is requested, use a representative sample with an explicit denominator and uncertainty. Do not tune findings to an expected rate such as 10–20%.

2. Discover through behavior

Read [behavioral-review.md](./references/behavioral-review.md) before auditing test bodies. For each reviewed test, trace:

1. **Claim:** what observable behavior does the name, fixture, or requirement promise? 2. **Real subject:** which production operation actually runs, including setup, imported helpers, autouse fixtures, module mocks, and dependency injection? 3. **Input:** does the fixture create the distinction the claim depends on? 4. **Oracle:** where does the expected answer come from, and which assertion checks it? 5. **Verdict:** will an incorrect result reach the runner as a failure? 6. **Counterexample:** what plausible wrong implementation would still pass this test?

Construct a concrete counterexample before declaring an assertion sufficient. Examples include ignoring one filter, selecting the first item instead of the minimum, dropping a forwarded prop, or returning an empty result where only the type is asserted. Prefer a changed observable result over a crash or syntax error.

Inspect both positive and negative tests. Trace assertion helpers to their actual checks. Read the relevant implementation and neighboring tests before reporting a gap. For mocks, identify whether the replaced operation owns the claimed behavior or is a collaborator of a real subject. Check arguments, call requirements, returned data, and observable effects at that boundary.

When delegation is available, split semantic discovery by owner or directory. Each reviewer discovers independently of detector output. Require exact locations, real subject, surviving regression, retained value, and evidence status. Independently verify the strongest findings before reporting them.

3. Use detectors as additional leads

Run each script by its path in this skill's directory, not the audited repository's `scripts/`, with the audited repository root as the working directory. Write output to a temporary directory. Use the repository's pinned tooling for runner commands. The scripts require Python 3.11+ and the standard library.

| Script | What its results mean | |---|---| | [weak_negatives.py](./scripts/weak_negatives.py) `<test globs>` | Heuristic candidates for ambiguous refusal assertions. Does not assess positive behavior or assertion data flow. | | [duplicate_tests.py](./scripts/duplicate_tests.py) `<test globs> --json` | Structural duplicate/consolidation candidates, not an effectiveness score. | | [junit_census.py](./scripts/junit_census.py) `<report.xml> --test-files <glob>` | Reported skips, possible collection gaps, and PHPUnit zero-assertion leads (adequate when not throwing is the documented contract). | | [guard_pins.py](./scripts/guard_pins.py