/test-coverage

name: test-coverage
description: Use after writing tests to assess coverage quality across structural, mutation, requirements, and API/integration dimensions; organized knowledge for choosing and interpreting coverage analyses.

Test Coverage Analysis

A reference manual for choosing, applying, and interpreting test-coverage analyses on an existing test suite.

This skill is a **knowledge reference**, not a procedure. It does not tell you when to write tests or which test types to design — that is the job of `design-testing-strategy`. It tells you, once tests exist, **which mechanical signal best measures what those tests do (and do not) exercise**, and how to read that signal honestly.

What Coverage Analysis Is

Test coverage analysis is the **post-hoc measurement** of how thoroughly a test suite exercises a software artifact along one or more **axes**. It answers the question *"what did my tests actually touch?"* — for some specific definition of "touch."

The word "coverage" is overloaded. It can mean any of:

• **Structural / code coverage** — which lines, statements, branches, conditions, or paths in the source code were executed (measured by instrumentation).
• **Mutation coverage** — what proportion of deliberately-injected source faults the test suite detects (measured by re-running the suite against mutated code).
• **Requirements / feature coverage** — which acceptance criteria, user stories, or specification clauses have at least one verifying test (measured by traceability).
• **API / integration coverage** — which endpoints, methods, status codes, contract interactions, and schema fields are exercised (measured by request/response inspection).
• **Specification-domain coverage** — equivalence classes, boundary values, parameter combinations, state transitions, error paths (measured by analyzing test inputs against a model).

Category correction: coverage is not a test type

Mutation testing, MC/DC, branch coverage, RTM linkage, contract coverage, and schema coverage are **measurements about** an existing test suite. They are **not** test types in the way unit, integration, e2e, contract, or smoke tests are.

• "Should I write a unit test or a mutation test?" is a malformed question. The correct framing is: *"I already have unit/integration tests; should I additionally run mutation analysis against them?"*
• Mutation tools generate variants of the source and re-execute the **existing** suite. They produce a score, not new tests.
• MC/DC and branch coverage are reports computed from instrumented runs of the **existing** suite.
• RTM linkage is a property of test metadata (tags, IDs), not a separate execution.

If a "test strategy" places mutation testing alongside unit / integration / e2e, that strategy has confused *what to test* with *how to measure the tests*. The two questions are orthogonal.

The asymmetry principle

**Low coverage is strong evidence of weak testing. High coverage is weak evidence of strong testing.**

Coverage is *necessary-but-not-sufficient*. A test can execute a line without asserting anything meaningful; 100% line coverage is routinely achievable with zero assertions ([thinkinglabs.io](https://thinkinglabs.io/articles/2022/03/19/the-fallacy-of-the-100-code-coverage.html), [codeintelligently.com](https://codeintelligently.com/blog/ai-generated-tests-false-confidence)). Use coverage as a **tripwire**, not a **trophy**. Once a coverage percentage becomes a target, it ceases to be a good metric (Goodhart's law applied to testing; see [Optivem Journal](https://journal.optivem.com/p/code-coverage-targets-recipe-for-disaster)).

What coverage analysis is NOT

• **NOT a measure of test quality.** Lines can execute without assertions.
• **NOT a measure of correctness.** Coverage proves the test ran, not that it would have failed on a bug.
• **NOT a synonym for "well tested".** Mutation testing routinely refutes 100%-coverage-with-no-assertions suites.
• **NOT a substitute for risk-based test selection** per [ISO/IEC/IEEE 29119](https://en.wikipedia.org/wiki/ISO/IEC_29119).
• **NOT a target.** Treat as a floor and a trend, never as the goal itself.

---

Per-Type Structure

Every coverage type in this skill is documented in the same six sub-fields, in this order:

1. **Definition** — what it measures. 2. **What it does NOT measure** — its limits / blind spots. 3. **Typical tools** — per ecosystem. 4. **When to use vs skip** — applicability heuristics. 5. **Targets / thresholds & pitfalls** — defensible numeric ranges (always with the risk caveat) and common gaming patterns. 6. **Cost-benefit ROI** — order-of-magnitude cost vs the signal you actually buy.

Scan any section by these headings.

---

Structural / Code Coverage

Measured by instrumenting the compiled or interpreted program and recording which structural elements (lines, statements, branches, conditions, paths) the test suite executes.

Line / Statement Coverage

• **Definition.** Percentage of source-code lines (or statements) executed at least once.
• **What it does NOT measure.** Whether branches were taken in both directions. Whether assertions verified the result. Whether boundary values were tested. Multiple statements on one line distort the metric ([Metridev](https://www.metridev.com/en/metrics/statement-vs-branch-coverage-understanding-the-difference/)).
• **Typical tools.**

| Ecosystem | Tool | |-----------|------| | JS/TS | [Istanbul / nyc](https://istanbul.js.org/) (built into Jest, Vitest, Karma); `--coverage` flag | | Python | [Coverage.py](https://coverage.readthedocs.io/) + `pytest-cov`; supports branch mode | | JVM | [JaCoCo](https://www.eclemma.org/jacoco/) — bytecode instrumentation, industry standard | | C/C++ | `gcov` / `lcov` / `gcovr`, [llvm-cov](https://llvm.org/docs/CommandGuide/llvm-cov.html) | | Go | `go test -cover`, `go tool cover` ([build-cover](https://go.dev/doc/build-cover) added integration-test mode in Go 1.20) |