/systematic-debugging

name: systematic-debugging
description: >
  Invoke BEFORE attempting any fix when a bug, test failure, error, warning,
  unexpected behavior, or performance regression appears. Enforces hypothesis-driven
  root cause analysis — no fix without evidence. Triggers on: error messages, stack
  traces, "it's broken", "not working", "bug", test failures, blank/white screens,
  "works locally but not in production", performance degradation, console warnings,
  "can't figure out why". Also routed by using-superpowers for debugging tasks.

Systematic Debugging

Do root-cause investigation before proposing fixes.

Iron Law

**No fix without evidence for root cause.**

This is non-negotiable. Every fix must trace back to a proven root cause. A fix applied without understanding why it works is not a fix — it is a mask.

Four Phases

Phase 0: Check Known Issues

• If `known-issues.md` exists at the project root, search it for the error message, error code, or failing test name.
• If a match is found, try the documented solution **first**. If it resolves the issue, stop — no further investigation needed.
• If no match or the documented solution doesn't work, proceed to Phase 1.

Phase 1: Investigate

• Read the **full** error output — not just the last line.
• Reproduce the bug reliably. If you cannot reproduce, you cannot fix. For tests that fail only in certain orderings (test pollution), run `bash find-polluter.sh <marker> <test-pattern>` from this skill's directory to identify which test is corrupting shared state. Read its exit code — they are not interchangeable:

| Exit | Meaning | |---|---| | 0 | Every matched test ran and none created the marker. A real clean result. | | 1 | Polluter found; it is named in the output. | | 2 | The pattern matched no test files. **Nothing ran, so nothing was proven** — fix the pattern and re-run. Never report this as clean. | | 3 | The marker already existed before the run. Reset it first; no test can be blamed until you do. |

• Check recent changes — what changed since it last worked?
• If `context-snapshot.json` exists at the project root: read it. The `changed_files` and `recent_commits` fields answer this immediately without additional git commands.
• Otherwise: run `git log --oneline -10` and `git diff HEAD~1..HEAD --name-only`.
• Add instrumentation (logging, breakpoints) at component boundaries.
• **Multi-component systems:** When the system has multiple components (CI → build → signing, API → service → database), add diagnostic logging at EACH component boundary before proposing fixes:

  For EACH component boundary:
    - Log what data enters the component
    - Log what data exits the component
    - Verify environment/config propagation
    - Check state at each layer

  Run once to gather evidence showing WHERE it breaks
  THEN analyze evidence to identify the failing component
  THEN investigate that specific component

Example (multi-layer system):

  # Layer 1: Workflow
  echo "=== Secrets available in workflow: ==="
  echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

  # Layer 2: Build script
  echo "=== Env vars in build script: ==="
  env | grep IDENTITY || echo "IDENTITY not in environment"

  # Layer 3: Signing script
  echo "=== Keychain state: ==="
  security list-keychains
  security find-identity -v

  # Layer 4: Actual signing
  codesign --sign "$IDENTITY" --verbose=4 "$APP"

This reveals which layer fails (e.g., secrets → workflow OK, workflow → build FAILS).

• Trace data/control flow backward from the error to its source.

Phase 2: Compare Patterns

• Find a known-good implementation in the repo that does something similar.
• Compare: behavior, config, assumptions, environment.
• List concrete differences — these are your investigation leads.

Phase 3: Hypothesize and Test

**Self-Consistency Gate** — Before committing to a hypothesis, apply multi-path reasoning (see `self-consistency-reasoner`):

1. Make 3-5 attempts at **the** root cause, each from a different starting point: trace forward from inputs, backward from the error, from recent changes, from similar past bugs. Each attempt names the one cause it arrives at — you are not brainstorming alternatives, you are checking whether different routes converge. 2. Group attempts that name the same cause, then take the majority. Report agreement as a count (e.g. 4/5):

• **Unanimous**: test the majority hypothesis.
• **Majority** (more than half, not all): test the majority hypothesis; name the minority one and test it next if the majority fails.
• **No majority** (half or fewer): **STOP.** Do not pick a hypothesis. The bug is ambiguous or multi-causal. Gather more evidence (add logging, reproduce under different conditions) before choosing a direction.

3. These attempts share your context and your blind spots, so agreement is weak evidence and disagreement is strong evidence. The smallest test below — not the vote — is what proves the hypothesis.

Then for the selected hypothesis:

• State it clearly before testing it.
• Run the **smallest** possible test/change to validate or disprove it.
• If disproven, discard it entirely and form a new hypothesis.
• **Never stack guesses.** One hypothesis at a time, tested in isolation.

Phase 4: Fix and Verify

• Add a failing regression test that reproduces the bug **first**.
• Apply **one** focused fix — never bundle multiple changes.
• Re-run the tests that cover the fix and confirm the issue is resolved. Scope per

`verification-before-completion` — a root cause that spanned subsystems earns the full suite; a one-module fix earns that module's suite.

Rationalization Table — Do Not Skip Investigation

| Temptation | Why it fails | |---|---| | "I know what the bug is" | You had a hypothesis, not proof. Test it. | | "Quick fix, investigate later" | Later never comes. The mask becomes permanent debt. | | "It's obviously this one line" | Obvious bugs get fix