/test-mutation

name: test-mutation
description: Mutation testing workflow. Systematically mutates source code to verify tests actually catch bugs. Multi-session with progress tracking.
model: opus

Test Mutate - Mutation Testing Workflow

Systematically introduces small changes (mutations) to source code, runs the test suite after each, and reports which mutations survive (tests don't catch them). Surviving mutations reveal genuine test coverage gaps that line coverage misses.

Philosophy

**Mutation score > line coverage.** A test that executes code but doesn't assert on results gives 100% line coverage and 0% mutation score. Mutation testing answers the real question: if a bug were introduced here, would the tests catch it?

**Multi-session by design.** Mutation testing is slow — each mutation requires a full test run. Progress is tracked in `.test-mutations.json` so you can work through a codebase incrementally across sessions.

**Autopilot by default.** After initial setup, the workflow runs unattended through all in-scope modules. It addresses all surviving mutations, commits after each module, and moves on. Human intervention is only needed during setup (scope selection, test command verification) and if an unrecoverable error occurs.

Workflow Overview

┌─────────────────────────────────────────────────────┐
│                  TEST MUTATE                        │
├─────────────────────────────────────────────────────┤
│  SETUP (interactive)                                │
│  1. Initialize (load or create tracking file)       │
│  2. Detect test command (first run only)            │
│  3. Determine scope (user selects, default: all)    │
│                                                     │
│  EXECUTION (autopilot — no user interaction)        │
│  For each module in scope:                          │
│    4. Spawn qa-test-mutator agent                   │
│    5. Update tracking file with results             │
│    6. Spawn SME to write tests for ALL survivors    │
│    7. Verify (new tests pass + re-mutate confirms)  │
│    8. Commit changes                                │
│  9. Final summary                                   │
└─────────────────────────────────────────────────────┘

Workflow Details

1. Initialize

Check for `.test-mutations.json` in the project root.

**If the file exists:**

• Load tracking data
• Show progress summary: X/Y modules tested, overall mutation score Z%
• List modules by status (completed, in-progress, pending)
• Proceed to step 3 (scope selection)

**If the file doesn't exist:**

• This is a first run — proceed to step 2 (test command detection)
• After detecting the test command, discover source files (see Module Discovery below)
• Create the tracking file with initial structure
• Ask user: "Should I commit `.test-mutations.json` to version control, or add it to `.gitignore`?"

2. Detect Test Command

**Try in order:**

1. `Makefile` with a `test` target → `make test` 2. `package.json` with a `test` script → `npm test` 3. `go.mod` present → `go test ./...` 4. `pyproject.toml` or `pytest.ini` or `setup.cfg` with pytest config → `pytest` 5. `Cargo.toml` → `cargo test` 6. `build.gradle` or `build.gradle.kts` → `gradle test`

**If none detected:** Ask the user: "What command runs your test suite?"

**Verify the command works** by running it once. If it fails, report the error and ask the user for the correct command.

Store the test command in the tracking file.

Module Discovery

Use Glob to find source files in the project. Exclude:

• Test files (`*_test.go`, `test_*.py`, `*.test.js`, `*.spec.ts`, etc.)
• Vendor/dependency directories (`vendor/`, `node_modules/`, `.venv/`, `target/`)
• Generated files (files with generation markers like `// Code generated`)
• Configuration files, documentation, assets

For each source file, attempt to identify covering test files using naming conventions:

• `auth.go` → `auth_test.go`
• `auth.py` → `test_auth.py` or `auth_test.py`
• `Auth.ts` → `Auth.test.ts` or `Auth.spec.ts`

Store discovered modules in the tracking file with status `pending`.

3. Determine Scope

Present the current state to the user:

## Mutation Testing Progress

Overall: 3/10 modules tested (mutation score: 87%)

### Completed
- src/auth.go — score: 100% (45 mutations)
- src/config.go — score: 92% (24 mutations, 2 survivors)

### In Progress
- src/payment.go — score: 80% (20/50 mutations tested)

### Pending
- src/api/handler.go
- src/models/user.go
- src/utils/parser.go
- ...

Scope? Enter file names/numbers, or press Enter to test all pending modules.

**User can:**

• Pick specific files by name or number (e.g., "1, 3, 5" or "src/auth.go")
• Resume an in-progress file
• Re-test a completed file (useful after adding tests)
• Press Enter / say "all" to test all pending modules (this is the default)

**Default:** All pending modules, processed in alphabetical order. If a module is in-progress, it is processed first.

**After scope is confirmed, the workflow enters autopilot mode. No further user interaction occurs until the run completes or an unrecoverable error is encountered.**

---

Steps 4-8 repeat for each module in scope (autopilot)

4. Spawn Mutator Agent

Spawn a `qa-test-mutator` agent with the selected source file and test command:

Apply mutation testing to the following source file:
- Source file: [path]
- Test command: [command]

Systematically mutate the source code, run tests after each mutation,
and report which mutations are killed vs survived.

Wait for the agent to complete and collect its results.

5. Update Tracking File

Parse the mutator agent's results and update `.test-mutations.json`:

• Set module status to `completed` (or `in_progress` if the agent reported partial coverage)
• Populate `mutations_by_type` with results grouped by mutation type
• Store surviving mutation examples in the `examples` arrays
• Calculate `mutation_score` for the module
• Update `glob